Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Create a new article
4197
Views
2
Helpful
1
Comments

LISP VXLAN Fabric Solution Design Guide

kadsteph
Cisco Employee
Cisco Employee

‎11-06-2023 11:24 AM - edited ‎11-08-2023 11:36 AM

 
 
07c1b4d3-f97c-4452-921b-ef298a7b80b1.png

 

 Solution Design Guide
Cisco Public



LISP VXLAN Fabric
Solution Design Guide

 

Contents

 

Authors

Devi Bellamkonda
Technical Marketing Engineer, Technical Leader

Kedar Karmarkar
Principal Technical Marketing Engineer


Document Organization

This document is organized into the following chapters:

Chapter Description
LISP/VXLAN Fabric Solution Components Key Components of the LISP/VXLAN fabric Solution
LISP/VXLAN Fabric Operational Planes Control Plane, Data Plane, Policy Plane Technologies
LISP/VXLAN Architecture Network Components Fabrics, Underlay Networks, Overlay Networks, and Shared Services
LISP/VXLAN Fabric Roles and Terminology Control Plane Node, Border Node, Edge Node, and other Fabric elements
LISP/VXLAN Design Considerations LAN Design Principles, Layer 3 Routed Access, Role Considerations, and Feature Considerations
LISP/VXLAN fabric Site Reference Models Site Size Reference Models and Topologies
Appendices Additional References and Resources



Icons Used in this Document.

 

Slide1.JPGSlide2.JPG

Slide3.JPG

Slide4.JPG

Slide5.JPG

Slide6.JPG

 

 

LISP/VXLAN Fabric

LISP/VXLAN fabric is the evolution from traditional campus designs to networks that directly implement the intent of an organization.

Fabric technology, provides wired and wireless campus networks with programmable overlays and easy-to-deploy network, permitting a physical network to host one or more logical networks to meet the design intent. In addition, fabric technology in the campus network enhances control of communications, providing segmentation and policy enforcement based on user identity and group membership. Segmentation is seamlessly integrated using Cisco Security Group, providing micro-segmentation for groups within a virtual network using Security Group tags (SGTs).

This design guide provides an overview of the requirements driving the evolution of campus network designs, followed by a discussion about the latest technologies and designs that are available for building a LISP/VXLAN fabric to address those requirements. The intended audience is a technical decision maker who wants to understand Cisco’s campus offerings, learn about the available technology options, and use leading practices for designing the best network for the needs of an organization.


LISP VXLAN Fabric Configuration Guide

More details on configuring the LISP/VXLAN fabric can be found at LISP VXLAN Fabric Configuration Guide, Cisco IOS XE Cupertino 17.9.x (Catalyst 9300 Switches)


LISP VXLAN Fabric Configuration via APIs

LISP/VXLAN Fabric Ansible Playbook

Automating SDA Fabric Creation with Ansible

Using Ansible LISP VXLAN Fabric YouTube Video


LISP VXLAN Fabric Troubleshooting Guide

More details on configuring the LISP/VXLAN fabric can be found at Troubleshoot LISP VXLAN Fabric on Catalyst 9000 Series Switches

 

Companion Resources

Find the companion guides related to LISP/VXLAN fabric automated via a controller Cisco Catalyst Center which is called Cisco SD-Access Cisco DNA Center & ISE Management Infrastructure Deployment Guide, SD-Access Fabric Provisioning Prescriptive Deployment Guide related deployment guides, design guides, and white papers, at the following pages:

SD-Access Resources on Cisco.com

Technology & Support Community.


Evolution of Campus Network Designs for Digital-Ready Organizations

With digitization, software applications are evolving from simply supporting business processes to becoming, in some cases, the primary source of business revenue and competitive differentiation. Organizations are now constantly challenged by the need to scale their network capacity to react quickly to application demands and growth. Because the campus network is used by people with different levels of access and their BYOD devices to access these applications, the wired and wireless LAN capabilities should be enhanced to support those changing needs.


Network Requirements for the Digital Organization

The following are the key requirements driving the evolution of existing campus networks.


Flexible Ethernet Foundation for Growth and Scale

  • Simplified deployment and automation—Network device configuration using APIs allows for very fast, lower-risk deployment of network devices and services.

  • Increased bandwidth needs—Bandwidth needs are doubling potentially multiple times over the lifetime of a network, resulting in the need for new networks to aggregate using 10 Gbps Ethernet to 40 Gbps to 100 Gbps capacities over time.

  • Increased capacity of wireless access points—The bandwidth demands on wireless access points (APs) with the latest 802.11ac Wave 2 and 802.11ax (Wi-Fi 6) technology now exceed 1 Gbps, and the IEEE has now ratified the 802.3bz standard that defines 2.5 Gbps and 5 Gbps Ethernet.

  • Additional power requirements from Ethernet devices—New devices, such as lighting, surveillance cameras, virtual desktop terminals, remote access switches, and APs, may require higher power to operate. The access layer design should have the ability to support Power over Ethernet (PoE) with 60W per port, offered with Cisco Universal Power Over Ethernet (UPOE), and the access layer should also provide PoE perpetual power during switch upgrade and reboot events. As power demands continue to increase with new endpoints, IEEE 802.3bt and Cisco UPOE-Plus (UPOE+) can provide power up to 90W per port.


Integrated Services and Security

  • Consistent wired and wireless security capabilities—Security capabilities, described below, should be consistent whether a user is connecting to a wired Ethernet port or connecting over the wireless LAN.

  • Identity services—Identifying users and devices connecting to the network provides the contextual information required to implement security policies for access control, network segmentation by using Security Group membership, and mapping of devices into virtual networks.

  • Network virtualization—The capability to share a common infrastructure while supporting multiple VNs with isolated data and control planes enables different sets of users and applications to be isolated securely.

  • Security Group policies—Creating access and application policies based on user group information provides a much easier and scalable way to deploy and manage security policies. Traditional access control lists (ACLs) can be difficult to implement, manage, and scale because they rely on network constructs such as IP addresses and subnets rather than group membership. Group membership is an IP-agnostic approach to policy creation which provides ease of operation for the network operator and a more scalable approach to ACLs.

  • Network segmentation—Security Group tags assigned from Security Group policies can be used to segment a network to achieve data plane isolation within physical and virtual networks.


LISP/VXLAN Solution Components

The LISP/VXLAN fabric solution is provided through a combination of hardware devices such as Layer 3 switches which support the fabric functionality and the Identity Services Engine (ISE), and wired and wireless device platforms which have fabric functionality.

 

Cisco Catalyst Center Hardware Appliance (Optional)

Cisco Catalyst Center software, run on Cisco Catalyst Center hardware appliance is available in form factors sized to support various functions Software image management (SWIM), PnP and new capabilities as they are available.

 

Tech tip

For additional information about the Cisco Catalyst Center Appliance capabilities, see the data sheet on Cisco.com.

 

Identity Services Engine (Optional)

Cisco Identity Services Engine (ISE) is a secure network access platform enabling increased management awareness, control, and consistency for users and devices accessing an organization’s network. ISE is an integral component of LISP/VXLAN fabric for implementing network access control policy. ISE performs policy implementation, enabling dynamic mapping of users and devices to Security Groups, and simplifying end- to-end security policy enforcement. Within ISE, users and devices are shown in a simple and flexible interface.

The LISP/VXLAN fabric solution integrates Cisco Security Group by supporting end-to-end Security Group Policy with Security Group Tags (SGTs). Security Group Tags are a metadata value that is transmitted in the header of fabric- encapsulated packets. This simplifies end-to-end security policy management and enforcement at a greater scale than traditional network policy implementations relying on IP access-lists.

 

ISE Personas

A Cisco ISE node can provide various services based on the persona that it assumes. Personas are simply the services and specific feature set provided by a given ISE node. The four primary personas are PAN, MnT, PSN, and pxGrid.

  • Policy Administration Node (PAN)— A Cisco ISE node with the Administration persona allows performs all administrative operations on Cisco ISE. It handles all system-related configurations that are related to functionality such as authentication, authorization, and auditing.

  • Monitor and Troubleshooting Node (MnT)— A Cisco ISE node with the Monitoring persona functions as the log collector and stores log messages from all the administration and Policy Service nodes in the network. This persona provides advanced monitoring and troubleshooting tools that used to effectively manage the network and resources. A node with this persona aggregates and correlates the data that it collects to provide meaningful information in the form of reports.

  • Policy Service Node (PSN)— A Cisco ISE node with the Policy Service persona provides network access, posture, guest access, client provisioning, and profiling services. This persona evaluates the policies and makes all the decisions. Typically, there would be more than one PSN in a distributed deployment. All Policy Service nodes that reside in the same high-speed Local Area Network (LAN) or behind a load balancer can be grouped together to form a node group.

  • Platform Exchange Grid (pxGrid)—A Cisco ISE node with pxGrid persona shares the context-sensitive information from Cisco ISE session directory with other network systems such as ISE ecosystem partner systems and Cisco platforms. The pxGrid framework can also be used to exchange policy and configuration data between nodes like sharing tags and policy objects. Security Group information like tag definition, value, and description can be passed from Cisco ISE to other Cisco management platforms such as Cisco Catalyst Center and Cisco Secure Network Analytics.

ISE supports standalone and distributed deployment models. Multiple, distributed nodes can be deployed together to provide failover resiliency and scale. The range of deployment options allows support for hundreds of thousands of endpoint devices. Minimally, a basic two-node ISE deployment is recommended for LISP/VXLAN fabric single site deployments with each ISE node running all services (personas) for redundancy.

LISP/VXLAN fabric nodes send authentication requests to the Policy Services Node (PSN) service persona running in ISE. In the case of a standalone deployment, the PSN persona is referenced by a single IP address.

Tech tip

For additional ISE deployment and scale details, please see ISE Performance & Scale on Cisco.com Security Community.


LISP/VXLAN Operational Planes

This chapter is organized into the following sections:

Chapter Section
LISP/VXLAN Operational Planes

Control Plane – LISP

Data Plane – VXLAN

Policy Plane – Cisco Security Group

There are three key technologies, which make up the LISP/VXLAN fabric solution, each performing distinct activities in different network planes of operation: control plane, data plane, policy plane.

  • Control Plane—Messaging and communication protocol between infrastructure devices in the fabric.

  • Data Plane—Encapsulation method used for the data packets.

  • Policy Plane—Used for security and segmentation.

In LISP/VXLAN fabric, the control plane is based on LISP (Locator/ID Separation Protocol), the data plane is based on VXLAN (Virtual Extensible LAN), the policy plane is based on Cisco Security Group.


Overlay Control Plane – LISP

In many networks, the IP address associated with an endpoint defines both its identity and its location in the network. In these networks, the IP address is used for both network layer identification (who the device is on the network) and as a network layer locator (where the device is at in the network or to which device it is connected). This is commonly referred to as addressing following topology. While an endpoint’s location in the network will change, who this device is and what it can access should not have to change. The Locator/ID Separation Protocol (LISP) allows the separation of identity and location though a mapping relationship of these two namespaces: an endpoint’s identity (EID) in relationship to its routing locator (RLOC).

The LISP control plane messaging protocol is an architecture to communicate and exchange the relationship between these two namespaces. This relationship is called an EID-to-RLOC mapping. This EID and RLOC combination provide all the necessary information for traffic forwarding, even if an endpoint uses an unchanged IP address when appearing in a different network location (associated or mapped behind different RLOCs).

Simultaneously, the decoupling of the endpoint identity from its location allows addresses in the same IP subnetwork to be available behind multiple Layer 3 gateways in disparate network locations (such as multiple wiring closets), versus the one-to-one coupling of IP subnetwork with network gateway in traditional networks. This provides the benefits of a Layer 3 Routed Access network, described in a later section, without the requirement of a subnetwork to only exist in a single wiring closet.

Instead of a typical traditional routing-based decision, the fabric devices query the control plane node to determine the routing locator associated with the destination address (EID-to-RLOC mapping) and use that RLOC information as the traffic destination. In case of a failure to resolve the destination routing locator, the traffic is sent to the default fabric border node. The response received from the control plane node is stored in the LISP map-cache, which is merged to the Cisco Express Forwarding (CEF) table and installed in hardware.


Overlay Data Plane – VXLAN

VXLAN is an encapsulation technique for data packets. When encapsulation is added to these data packets, a tunnel network is created. Tunneling encapsulates data packets from one protocol inside a different protocol and transports the original data packets, unchanged, across the network. A lower-layer or same-layer protocol (from the OSI model) can be carried through this tunnel creating an overlay. In LISP/VXLAN, this overlay network is referred to as the fabric.

VXLAN is a MAC-in-IP encapsulation method. It provides a way to carry lower-layer data across the higher Layer 3 infrastructure. Unlike routing protocol tunneling methods, VXLAN preserves the original Ethernet header from the original frame sent from the endpoint. This allows for the creation of an overlay at Layer 2 and at Layer 3 depending on the needs of the original communication. For example, Wireless LAN communication (IEEE 802.11) uses Layer 2 datagram information (MAC Addresses) to make bridging decisions without a direct need for Layer 3 forwarding logic.

LISP/VXLAN fabric also places additional information in the fabric VXLAN header including alternative forwarding attributes that can be used to make policy decisions by identifying each overlay network using a VXLAN network identifier (VNI). Layer 2 overlays are identified with a VLAN to VNI correlation (L2 VNI), and Layer 3 overlays are identified with a VRF to VNI correlation (L3 VNI).

Any encapsulation method is going to create additional MTU (maximum transmission unit) overhead on the original packet. As show in Figure 2, VXLAN encapsulation uses a UDP transport. Along with the VXLAN and UDP headers used to encapsulate the original packet, an outer IP and Ethernet header are necessary to forward the packet across the wire. At minimum, these extra headers add 50 bytes of overhead to the original packet.

Figure 1. Fabric VXLAN (VNI) Encapsulation Overhead

image8.png

 

Policy Plane – Cisco Security Group

Cisco Security Group decouples access that is based strictly on IP addresses and VLANs by using logical groupings in a method known as Security Group Policy. The goal of Cisco Security Group is to assign an SGT value to the packet at its ingress point into the network.An access policy is enforced at the egress point in the network based on this tag information

.

An SGT is a form of metadata and is a 16-bit value assigned by ISE in an authorization policy when user, device, or application connects to the network.

The fabric VXLAN encapsulation method is used by both the data plane and policy plane. In the policy plane, the alternative forwarding attributes (the SGT value and VRF values) are encoded into the header and carried across the overlay.

Figure 2. Fabric VXLAN Alternative Forwarding Attributes

image9.png

 

 

Tech tip

A bit-level diagram of the VXLAN encapsulation method used in LISP/VXLAN fabric along with low-level details on policy constructs insertion into the header can be found in Appendix A.


LISP/VXLAN Architecture Network Components

This chapter is organized into the following sections:

Chapter Section
LISP/VXLAN Architecture Network Components

What is a Fabric?

Underlay Network

Overlay Network

Shared Services

The LISP/VXLAN architecture is supported by fabric technology implemented for the campus, enabling the use of virtual networks (overlay networks) running on a physical network (underlay network) creating alternative topologies to connect devices. This section describes and defines the word fabric, discusses the LISP/VXLAN fabric underlay and overlay network, and introduces shared services which are a shared set of resources accessed by devices in the overlay. This section provides an introduction for these fabric-based network terminologies used throughout the rest of the guide.


What is a Fabric?

A fabric is simply an overlay network. Overlays are created through encapsulation, a process which adds additional header(s) to the original packet or frame. An overlay network creates a logical topology used to virtually connect devices that are built over an arbitrary physical underlay topology. In an idealized, theoretical network, every device would be connected to every other device. In this way, any connectivity or topology imagined could be created. While this theoretical network does not exist, there is still a technical desire to have all these devices connected to each other in a full mesh. This is where the term fabric comes from: it is a cloth where everything is connected. In networking, an overlay (or tunnel) provides this logical full-mesh connection.


Underlay Network

The underlay network is defined by the physical switches and routers that are used to deploy the LISP/VXLAN fabric network. All network elements of the underlay must establish IP connectivity via the use of a routing protocol. In LISP/VXLAN fabric, the underlay switches (edge nodes) support the physical connectivity for users and endpoints. However, end-user subnets and endpoints are not part of the underlay network—they are part of the overlay network.

Figure 3. Overlay and Underlay Relationship

Slide8.JPG

 

 

 

 

Overlay Network

An overlay network is created on top of the underlay network through virtualization (virtual networks). The data plane traffic and control plane signaling are contained within each virtualized network, maintaining isolation among the networks and an independence from the underlay network. Multiple overlay networks can run across the same underlay network through virtualization. In LISP/VXLAN fabric, the user-defined overlay networks are provisioned as a virtual routing and forwarding (VRF) instance that provide separation of routing tables.

LISP/VXLAN fabric allows for the extension of Layer 2 and Layer 3 connectivity across the overlay services provided by LISP. Layer 2 overlay services emulate a LAN segment to transport Layer 2 frames by carrying a subnet over the Layer 3 underlay as shown in Figure 4.

Figure 4. Layer 2 Overlay – Logically Switch Connectivity

Slide9.JPG

 

 

 

 

 

Layer 3 overlays abstract the IP-based connectivity from the physical connectivity as shown in Figure 5. This can allow multiple IP networks to be part of each virtual network. Each Layer 3 overlay, its routing tables, and its associated control planes are completely isolated from each other.

Figure 5. Layer 3 Overlay – Logically Routed Connectivity

Slide10.JPG

 

 

 

 

The following diagram shows an example of two subnets that are part of the overlay network. The subnets stretch across physically separated Layer 3 devices–two edge nodes. The RLOC interfaces, or Loopback 0 interfaces in LISP/VXLAN fabric, are the only underlay routable address that are required to establish connectivity between endpoints of the same or different subnet within the same VN.

Figure 6. Subnet Stretching – Example

Slide11.JPG

 

 

 

Shared Services

Shared services refer to the elements that are employed across multiple virtual networks, providing a common infrastructure that ensures efficient utilization of resources.Shared services can take various forms including wireless infrastructure, DHCP, DNS, IP Address Management(IPAM), Active Directory (AD), Internet access, NTP Servers,Building Management Systems (BMS), servers, and critical systems.To secure these shared services, the application of a VRF-Aware Peer attached directly outside of the fabric enables route leaking of shared service prefixes across multiple networks. Moreover, firewalls add an extra layer of security and allow traffic monitoring between virtual networks.


LISP/VXLAN Fabric Roles and Terminology

This chapter is organized into the following sections:

Chapter Section
LISP/VXLAN Fabric Roles and Terminology

Control Plane Node

Edge Node

Intermediate Node

Border Node

Fabric in a Box

Fabric WLC

Fabric-Mode Access Point

Fabric Embedded Wireless.

Transit and Peer Networks

The LISP/VXLAN fabric solution is provided through a combination of Identity Services Engine (ISE), and wired and wireless device platforms which have fabric functionality. The wired and wireless device platforms are utilized to create the elements of a fabric site. A fabric site is defined as location that has its own Border, control plane node and an edge node. For wireless, a fabric-mode WLC is dedicated to the site, and for policy, an ISE Policy Service Node (PSN) is used. A fabric border node is required to allow traffic to egress and ingress the fabric site.

A fabric role is a LISP/VXLAN software construct running on physical hardware. These software constructs were designed with modularity and flexibility in mind. For example, a device can run a single role, or a device can also run multiple roles. 

Figure 7. LISP/VXLAN Fabric Roles - Example

Slide12.JPG

 

 

 

 

Control Plane Node

The LISP/VXLAN fabric control plane node is based on the LISP Map Server and Map Resolver functionality combined on the same node. The control plane node’s database tracks all endpoints in the fabric site and associates the endpoints to fabric nodes, decoupling the endpoint IP address or MAC address from the location (closest device) in the network.

The control plane node enables the following functions:

  • Host tracking database—The host tracking database (HTDB) is a central repository of Endpoint ID to Routing Locator (EID-to-RLOC) bindings where the RLOC is simply the IP address of the Loopback 0 interface on a fabric node. The HTDB is equivalent to a LISP site, in traditional LISP, which includes what endpoint ID can be and have been registered.

  • Endpoint identifiers (EID)—The endpoint identifier is an address used for numbering or identifying an endpoint device in the network. The LISP/VXLAN fabric solution supports MAC Address, IPv4 Address, and IPv6 addresses as EIDs.

  • Map Server—The LISP Map Server (MS) receives endpoint registrations indicating the associated RLOC and uses this to populate the HTDB.

  • Map Resolver—The LISP Map Resolver (MR) responds to queries from fabric devices requesting RLOC mapping information from the HTDB in the form of an EID-to-RLOC binding. This tells the requesting device to which fabric node an endpoint is connected and thus where to direct traffic.


Edge Node

The LISP/VXLAN fabric edge nodes are the equivalent of an access layer switch in a traditional campus LAN design. The edge node functionality is based on the Ingress and Egress Tunnel Routers (xTR) in LISP. The edge nodes must be implemented using a Layer 3 routed access design. The provide the following fabric functions:

  • Endpoint registration—Each edge node has a LISP control plane session to all control plane nodes. After an endpoint is detected by the edge node, it is added to a local database called the EID-table. Once the host is added to this local database, the edge node also issues a LISP map-register message to inform the control plane node of the endpoint so the central HTDB is updated.

  • Anycast Layer 3 gateway—A common gateway (IP and MAC addresses) is used at every edge node that shares a common EID subnet providing optimal forwarding and mobility across different RLOCs. On edge nodes, the Anycast Layer 3 gateway is instantiated as a Switched Virtual Interface (SVI) with a hard- coded MAC address that is uniform across all edge nodes within a fabric site.

  • Mapping of user to virtual network—Endpoints are placed into virtual networks by assigning the endpoint to a VLAN associated to an SVI that is forwarding for a VRF. Together, these make up the Layer 2 and Layer 3 LISP VNIs, respectively, which maintain fabric segmentation even at the control plane communication level.

  • AAA Authenticator—The mapping of endpoints into VLANs can be done statically or dynamically using an Authentication Server. Operating as a Network Access Device (NAD), the edge node is an integral part of the IEEE 802.1X port-based authentication process by collecting authentication credentials from connected devices, relaying the to the Authentication Server, and enforcing the authorization result.

  • VXLAN encapsulation/de-encapsulation—Packets and frames received from endpoint, either directly connected to an edge node or access point, are encapsulated in fabric VXLAN, and forwarded across the overlay. Traffic is either sent to another edge node or to the border node, depending on the destination.

When fabric encapsulated traffic is received for the endpoint, such as from a border node or from another edge node, it is de-encapsulated and sent to that endpoint. This encapsulation and de-encapsulation of traffic enables the location of an endpoint to change, as the traffic can be encapsulated towards different edge nodes in the network, without the endpoint having to change its address.


Intermediate Node

Intermediate nodes are part of the Layer 3 network used for interconnections among the devices operating in a fabric role such as the interconnections between border nodes and edge nodes. These interconnections are created in the Global Routing Table on the devices and is also known as the underlay network. For example, if a three-tier campus deployment provisions the core switches as the border nodes and the access switches as the edge nodes, the distribution switches are the intermediate nodes.

The number of intermediate nodes is not limited to a single layer of devices. For example, borders nodes may be provisioned on an enterprise edge switch resulting in the intermediate nodes being the core and distribution layers as shown in Figure 8.

Figure 8. Intermediate Nodes at Core and Distribution in LISP/VXLAN fabric – Example

Slide13.JPG

 

 

 

 

Intermediate nodes do not have a requirement for VXLAN encapsulation/de-encapsulation, LISP control plane messaging support, or SGT awareness. Their requirement is to provide IP reachability, physical connectivity, and to support the additional MTU requirement to accommodate the larger-sized IP packets encapsulated with fabric VXLAN information. Intermediate nodes simply route and transport IP traffic between the devices operating in fabric roles.

Tech tip

VXLAN adds 50 bytes to the original packet. The common denominator and recommended MTU value available on devices operating in a fabric role is 9100. Network should have a minimum starting MTU of at least 1550 bytes to support the fabric overlay. MTU values between 1550 and 9100 are supported along with MTU values larger than 9100 though there may be additional configuration and limitations based on the original packet size.


Border Node

The fabric border nodes serve as the gateway between the LISP/VXLAN fabric site and the networks external to the fabric. The border node is responsible for network virtualization interworking and SGT propagation from the fabric to the rest of the network.

Border nodes implement the following functions:

  • Advertisement of EID subnets—BGP (Border Gateway Protocol) is the routing protocol configured to advertise the coarse-aggregate endpoint prefix space outside the fabric. This is also necessary so that traffic from outside of the fabric destined for endpoints in the fabric is attracted back to the border nodes.

  • Fabric site exit point—The external border node is the gateway of last resort for the fabric edge nodes. This is implemented using LISP Proxy Tunnel Router (PxTR) functionality. Also possible is the internal border node which registers known networks (IP subnets) with the fabric control plane node.

  • Network virtualization extension to the external world—The border node can extend network virtualization from inside the fabric to outside the fabric by using VRF-lite and VRF-aware routing protocols to preserve the segmentation.

  • Policy mapping—The Border Node can optionally map SGT information, if the users choose to, but it does not map by default. Discussed further in the Micro-segmentation section, when the fabric packet is de-encapsulated at border, SGT information can be propagated using SGT Exchange Protocol (SXP) or by directly mapping SGTs into the Cisco metadata field in a packet using inline tagging.

  • VXLAN encapsulation/de-encapsulation—Packets and frames received from outside the fabric and destined for an endpoint inside of the fabric are encapsulated in fabric VXLAN by the border node. Packets and frames sourced from inside the fabric and destined outside of the fabric are de-encapsulated by the border node. This is similar to the behavior used by an edge node except, rather than being connected to endpoints, the border node connects a fabric site to a non-fabric network.


Fabric in a Box

Fabric in a Box is a LISP/VXLAN fabric construct where the border node, control plane node, and edge node are running on the same fabric node. This may be a single switch, a switch with hardware stacking, or a StackWise Virtual deployment. Fabric in a Box is discussed further in Fabric in a Box Site Reference Model section.


Fabric WLC

Both fabric WLCs and non-fabric WLCs provide AP image and configuration management, client session management, and mobility services. Fabric WLCs provide additional services for fabric integration such as registering MAC addresses of wireless clients into the host tracking database of the fabric control plane nodes during wireless client join events and supplying fabric edge node RLOC-association updates to the HTDB during client roam events.

In a traditional Cisco Unified Wireless network, or non-fabric deployment, both control traffic and data traffic are tunneled back to the WLC using CAPWAP (Control and Provisioning of Wireless Access Points). From a CAPWAP control plane perspective, AP management traffic is generally lightweight, and it is the client data traffic that is generally the larger bandwidth consumer. Wireless standards have allowed larger and larger data rates for wireless clients, resulting in more and more client data that is tunneled back to the WLC. The requires a larger WLC with multiple high-bandwidth interfaces to support the increase in client traffic.

In non-fabric wireless deployments, wired and wireless traffic have different enforcement points in the network. Quality of service and security are addressed by the WLC when it bridges the wireless traffic onto the wired network. For wired traffic, enforcement is addressed by the first-hop access layer switch. This paradigm shifts entirely with LISP/VXLAN fabric Wireless. In LISP/VXLAN fabric Wireless, the CAPWAP tunnels between the WLCs and APs are used for control traffic only. Data traffic from the wireless endpoints is tunneled to the first-hop fabric edge node where security and policy can be applied at the same point as with wired traffic.

Typically, fabric WLCs connect to a shared services network though a distribution block or data center network that is connected outside the fabric, and the WLC management IP address exists in the global routing table. For wireless APs to establish a CAPWAP tunnel for WLC management, the APs must be in a VN that has access to this external device. This means that the APs are deployed in the global routing table and that the WLC’s address must be present in the GRT within the fabric site.

In the LISP/VXLAN fabric solution, wireless APs will reside within an overlay VN named default instance of LISP which maps to the global routing table. This avoids the need for route leaking (a multi-VRF device selectively sharing routing information) to establish connectivity between the WLCs and the APs. Each fabric site must have a WLC unique to that site. Most deployments place the WLC in the local fabric site itself, not across a WAN, because of latency requirements for fabric mode APs. Further latency details are covered in the section below.

Tech tip

Strategies on connecting the fabric to shared services and details on route leaking and fusion routing are discussed in the External Connectivity and VRF-Aware Peer sections below.

Fabric access points operate in fabric mode. This requires an RTT (round-trip time) of 20ms or less between the AP and the WLC. This generally means that the WLC is deployed in the same physical site as the access points.


Fabric-Mode Access Points

The fabric-mode APs are Cisco Wi-Fi 6 and Wi-Fi 6E (802.11ax) and 802.11ac Wave 2 APs associated with the fabric WLC that have been configured with one or more fabric-enabled SSIDs. Fabric APs establish a CAPWAP control plane tunnel to the fabric WLC and join as fabric mode APs. They must be directly connected to the fabric edge node in the fabric site. For their data plane, Fabric APs establish a VXLAN tunnel to their first-hop fabric edge switch where wireless client traffic is terminated and placed on the wired network.

Fabric APs are considered a special case wired host. As a wired host, access points have a dedicated EID-space and are registered with the control plane node. This EID-space is associated with a predefined overlay network called default instance of LISP. It is a common EID-space (prefix space) and common virtual network for all fabric APs within a fabric site. The assignment to this overlay virtual network allows management simplification by using a single subnet to cover the AP infrastructure at a fabric site.


Fabric Embedded Wireless

To enable wireless controller functionality without a hardware WLC in distributed branches and small campuses, the Cisco Catalyst 9800 Embedded Wireless Controller is available for Catalyst 9000 Series switches as a software package on switches running in Install mode. The wireless control plane of the embedded controller operates like a hardware WLC. CAPWAP tunnels are initiated on the APs and terminate on the Cisco Catalyst 9800 Embedded Wireless Controller. The data plane uses VXLAN encapsulation for the overlay traffic between the APs and the fabric edge node.

The Catalyst 9800 Embedded Wireless Controller for Catalyst 9000 Series switches is supported for LISP/VXLAN fabric deployments with two topologies:

  • Cisco Catalyst 9000 Series switches functioning as co-located border and control plane.

  • Cisco Catalyst 9000 Series switches functioning as a Fabric in a Box.

Figure 9. LISP/VXLAN Embedded Wireless Supported Topologies

Slide15.JPG

 

 

 

 

Transit and Peer Network

Transits, referred to as Transit/Peer Networks configuration will reside on the border node to connect a fabric site to the external world. This connectivity may be MAN, WAN, or Internet. The WAN could be MPLS, SD-WAN, IWAN, or other WAN variations. We refer to the transits as IP-Based transit/peer network.

  • IP-Based Transits—Packets are de-encapsulated from the fabric VXLAN into native IP. Once in native IP, they are forwarded using traditional routing and switching modalities. IP-based transits are provisioned with VRF-lite to connect to the upstream device. IP-Based transits are commonly used to connect to shared services using a VRF-Aware Peer and connecting to upstream routing infrastructure or firewall for connectivity to WAN and Internet.


Fabric Site

A fabric site is composed of a unique set of devices operating in a fabric role along with the intermediate nodes used to connect those devices. At minimum, a fabric site must have a control plane node and an edge node, and to allow communication to other destinations outside of the fabric site, a border node. A fabric site generally has an associated WLC and potentially an ISE Policy Service Node (PSN).


LISP/VXLAN Design Considerations

This chapter is organized into the following sections:

Chapter Section
LISP/VXLAN Fabric Design Considerations

LAN Design Principles Device Role

Design Principles

Feature-Specific Design Requirements

Wireless Design

External Connectivity

Security Policy Considerations

Multidimensional Considerations

Any successful design or system is based on a foundation of solid design theory and principles. Designing a LISP/VXLAN fabric network or fabric site as a component of the overall enterprise LAN design model is no different than designing any large networking system. The use of a guiding set of fundamental engineering principles ensures that the design provides a balance of availability, security, flexibility, and manageability required to meet current and future technology needs.

This section provides design guidelines that are built upon these balanced principles to allow a LISP/VXLAN fabric network architect to build the fabric using next-generation products and technologies. These principles allow

for simplified application integration and the network solutions to be seamlessly built on a modular, extensible, and highly available foundation design that can provide continuous, secure, and deterministic network operations.

This section will begin by discussing LAN design principles, discusses design principles covering specific device roles, feature-specific design considerations, wireless design, external connectivity, security policy design, and multidimensional considerations.


LAN Design Principles

This major section is organized into the following subsections:

 

Section Subsection
LAN Design Principles

Underlay Network Design Overlay Network Design Shared Services Design

Fabric DHCP Overview and Design

Latency

 

The following LAN design principles apply to networks of any size and scale. This section looks at underlay network, overlay network, shared services, and services blocks, DHCP in the Fabric along with latency requirements for the network.


Underlay Network Design

This section is organized into the following subsections:

Section Subsection
Underlay Network Design

Layer 3 Routed Access Introduction

Enterprise Campus Architecture

Layer 2 (Switched) Access – Traditional Campus Design

About Layer 3 Routed Access

Layer 3 Routed Access and LISP/VXLAN fabric Network Design

StackWise Virtual in LISP/VXLAN and Layer 3 Routed Access

Networks

Having a well-designed underlay network ensures the stability, performance, and efficient utilization of the LISP/VXLAN fabric network.

While deploying the underlay networks for the fabric, have the following general design requirements:

  • Layer 3 Routed Access—The use of a Layer 3 routed access network for the fabric provides the highest level of availability without the need to use loop avoidance protocols such as Spanning-Tree (STP), interface bundling techniques using link aggregation technologies such as EtherChannel, and Layer 2 redundancy technologies like StackWise Virtual (SVL), Virtual Switching System (VSS), or Nexus Virtual Port-Channels (vPCs).

  • Increase default MTU—The VXLAN header adds 50 bytes of encapsulation overhead. Enabling a campus and branch wide MTU of 9100 ensures that Ethernet jumbo frames can be transported without fragmentation inside the fabric.

  • Point-to-point links—Point-to-point links provide the quickest convergence times because they eliminate the need to wait for the upper layer protocol timeouts typical of more complex topologies. Combining point-to-point links with the recommended physical topology design provides fast convergence in the event of a link failure.

  • The fast convergence is a benefit of quick link failure detection triggering immediate use of alternate topology entries preexisting in the routing and forwarding table. Implement the point-to-point links using optical technology as optical (fiber) interfaces are not subject to the same electromagnetic interference (EMI) as copper links. Copper interfaces can be used, though optical ones are preferred.

  • ECMP—Equal-cost multi-path routing is a routing strategy where next-hop packet forwarding to a single destination can occur over multiple best paths. Load balancing between these ECMP paths is performed automatically using Cisco Express Forwarding (CEF). ECMP-aware routing protocols should be used to take advantage of the parallel-cost links and to provide redundant forwarding paths for resiliency.

  • BFD—Bidirectional Forwarding Detection enhances fault detection and convergence characteristics of routing protocols. Routing protocols use the absence of Hello packets to determine if an adjacent neighbor is down (commonly called Hold Timer or Dead Timer). Thus, the ability to detect liveliness in a neighbor is based on the frequency of Hello packets.

  • Each Hello packet is processed by the routing protocol adding to the overhead and rapid Hello messages creates an inefficient balance between liveliness and churn. BFD provides low-overhead, sub-second detection of failures in the forwarding path between devices and can be set a uniform rate across a network using different routing protocols that may have variable Hello timers.

  • NSF—Non-stop forwarding, or graceful restart, works with SSO (stateful switchover) to provide continued forwarding of packets in the event of a route processor (RP) switchover. NSF-aware IGP routing protocols should be used to minimize the amount of time that a network is unavailable following a switchover.

  • SSO—Stateful Switchover maintains stateful feature information, such as user session, by synchronizing state information between a primary and backup route processor such as an RPs in routing platforms or supervisor engines in switching platforms. SSO should be enabled in concert with NSF on supported devices.

  • Loopback propagation—The loopback addresses assigned to the underlay devices need to propagate outside of the fabric to establish connectivity to infrastructure services such as fabric control plane nodes, DNS, DHCP, and AAA.

  • Loopback 0 interfaces (RLOC) require a /32 subnet mask. These addresses also be propagated throughout the fabric site.Reachability between loopback address (RLOCs) cannot use the default route. They must use a /32 route.

  • WLC reachability—Connectivity to the WLC should be treated like reachability to the loopback addresses. A default route in the underlay cannot be used by the APs to reach the WLCs. A specific route (non- default route) to the WLC IP address must exist in the Global Routing Table at each switch where the APs are physically connected. This can be a host route (/32) or summarized route.


Layer 3 Routed Access Introduction

For campus designs requiring simplified configuration, common end-to-end troubleshooting tools, and the fastest convergence, a design using Layer 3 switches in the access layer (routed access) in combination with Layer 3 switching at the distribution layer and core layers provides the most rapid convergence of data and control plane traffic flows.

This section describes the Enterprise Campus hierarchical network structure followed by traditional campus designs that use the distribution layer as the Layer 2/Layer 3 boundary (switched access). This traditional design is then contrasted against moving the Layer 2/Layer 3 boundary to the access layer (routed access), a requirement for LISP/VXLAN fabric, and finally discusses design considerations for Layer 3 routed access.


Enterprise Campus Architecture Introduction

Hierarchical network models are the foundation for modern network architectures. This allows network systems, both large and small, simple, and complex, to be designed and built using modularized components. These components are then assembled in a structured and hierarchical manner while allowing each piece (component, module, and hierarchical point) in the network to be designed with some independence from overall design. Modules (or blocks) can operate semi-independently of other elements, which in turn provides higher availability to the entire system. By dividing the Campus system into subsystems and assembling them into a clear order, a higher degree of stability, flexibility, and manageability is achieved for the individual pieces of the network and the campus deployment.

These hierarchical and modular networks models are referred to as the Cisco Enterprise Architecture Model and have been the foundation for building highly available, scalable, and deterministic networks for nearly two decades. The Enterprise Architecture Model separates the network into different functional areas called modules or blocks designed with hierarchical structures. The Enterprise Campus is traditionally defined with a three-tier hierarchy composed of the Core, Distribution, and Access Layers. In smaller networks, two-tiers are common with core and distribution collapsed into a single layer (collapsed core). The key idea is that each element in the hierarchy has a specific set of functions and services that it offers. The same key idea is referenced later in the fabric control plane node and border node design section.

The access layer represents the network edge where traffic enters or exits the campus network towards users, devices, and endpoints. The primary function of an access layer switch is to provide network access to the users and endpoint devices such as PCs, printers, access points, telepresence units, and IP phones.

The distribution layer is the interface between the access and the core providing multiple, equal cost paths to the core, intelligent switching and routing, and aggregation of Layer 2 and Layer 3 boundaries.

The Core layer is the backbone interconnecting all the layers and ultimately providing access to the compute and data storage services located in the data center and access to other services and modules throughout the network. It ties the Campus together with high bandwidth, low latency, and fast convergence.

Tech tip

For additional details on the Enterprise Campus Architecture Model, please see:


Layer 2 (Switched) Access – Traditional Campus Design

In typical hierarchical design, the access layer switch is configured as a Layer 2 switch that forwards traffic on high-speed trunk ports to the distribution switches. The distribution switches are configured to support both Layer 2 switching on their downstream trunks and Layer 3 switching on their upstream ports towards the core of the network. The function of the distribution switch in this design is to provide boundary functions between the bridged Layer 2 portion of the campus and the routed Layer 3 portion, including support for the default gateway, Layer 3 policy control, and all required multicast services.

The distribution block would typically span VLANs across the layer with the default gateway provided through SVI (Switched Virtual Interfaces) and distribution peer switches running first-hop redundancy protocols (FHRP) such as HSRP (Hot Standby Router Protocol). Alternatively, distribution switch peers may run Virtual Switching System (VSS) or Stackwise Virtual (SVL) to act as a single, logical entity and provide Multi-chassis EtherChannel (MEC) to access layer switches.

Layer 2 access networks provide the flexibility to allow applications that require Layer 2 connectivity to extend across multiple wiring closets. This design does come with the overhead of Spanning-Tree Protocol (STP) to ensure loops are not created when there are redundant Layer 2 paths in the network.

The stability of and availability for the access switches is layered on multiple protocol interactions in a Layer 2 switched access deployment. For example, in a common Layer 2 access network, the HSRP gateway for a VLAN should be the STP root bridge. Link Aggregation (LAG) is provided via LACP (Link Aggregation Control Protocol) or PAgP (Port Aggregation Protocol) to connect to upstream switches using MEC. These upstream switches are often configured with VSS / SVL, separate protocols themselves from LAG, to provide a logical entity across two physical devices. Trunking protocols ensure VLANs are spanned and forwarded to the proper switches throughout the system. While all of this can come together in an organized, deterministic, and accurate way, there is much overhead involved both in protocols and administration, and ultimately, spanning- tree is the protocol pulling all the desperate pieces together. All the other protocols and their interactions rely on STP to provide a loop-free path within the redundant Layer 2 links. If a convergence problem occurs in STP, all the other technologies listed above can be impacted.


About Layer 3 Routed Access

The hierarchical Campus, whether Layer 2 switched or Layer 3 routed access, calls for a full mesh equal-cost routing paths leveraging Layer 3 forwarding in the core and distribution layers of the network to provide the most reliable and fastest converging design for those layers. An alternative to Layer 2 access model described above is to move the Layer 3 demarcation boundary to the access layer. Layer 2 uplink trunks on the Access

switches are replaced with Layer 3 point-to-point routed links. This brings the advantages of equal cost path routing to the Access layer.

Using routing protocols for redundancy and failover provides significant convergence improvement over spanning-tree protocol used in Layer 2 designs. Each switch has two routes and two associated hardware Cisco Express Forwarding (CEF) forwarding adjacency entries. Traffic is forwarded with both entries using equal-cost multi-path (ECMP) routing. In the event of a failure of an adjacent link or neighbor, the switch hardware and software immediately remove the forwarding entry associated with the lost neighbor. However, the switch still has a remaining valid route and associated CEF forwarding entry. With an active and valid route, traffic is still forwarded. The result is a simpler overall network configuration and operation, dynamic load balancing, faster convergence, and a single set of troubleshooting tools such as ping and traceroute.

Tech tip

Layer 3 routed access is defined by Layer 3 point-to-point routed links between devices in the Campus hierarchy. In a LISP/VXLAN fabric network, Access and distribution switch should not peer with their upstream neighbors using SVIs and trunk ports. SVIs and trunk ports between the layers still have an underlying reliance on Layer 2 protocol interactions.


Layer 3 Routed Access and LISP/VXLAN fabric Network Design

LISP/VXLAN fabric networks start with the foundation of a well-design, highly available Layer 3 routed access foundation. For optimum convergence at the core and distribution layer, build triangles, not squares, to take advantage of equal-cost redundant paths for the best deterministic convergence. In Figure 10, the graphic on the left shows triangle topologies which are created by devices crosslinking with each other and with their upstream/downstream peers. The graphic on the right shows square topologies that are created when devices are not connected to both upstream/downstream peers. Square topologies should be avoided.

Figure 10. Triangle vs. Square Topologies

Slide16.JPG

 

 

As illustrated in Figure 11, Core switch peer devices should be cross linked to each other. Distribution switches within the same distribution block should be crosslinked to each other and connected to each core switch.

Access switches should be connected to each distribution switch within a distribution block, though they do not need to be cross-linked to each other.

Figure 11. Layer 3 Routed Access Design

Slide17.JPG

 

 

The interior gateway routing (IGP) routing protocol should be fully featured and support Non-Stop Forwarding, Bidirectional Forwarding Detection, and equal cost multi-path. IS-IS, EIGRP, and OSPF each support these features and can be used as an IGP to build a Layer 3 routed access network. Point-to-point links should be optimized with BFD, a hard-coded carrier-delay and load-interval, enabled for multicast forwarding, and CEF should be optimized to avoid polarization and under-utilized redundant paths.

Tech tip

For more information on Layer 3 routed access design methodology and high availability tuning, please see: Routed Access Layer Design Guide, Tuning for Optimized Convergence Guide , and Routed Access Layer Assurance Guide.


StackWise Virtual in LISP/VXLAN fabric and Layer 3 Routed Access Networks

StackWise Virtual (SVL), like its predecessor Virtual Switching System (VSS), is designed to address and simplify Layer 2 operations. It is the virtualization of two physical switches into a single logical switch from a control and management plane perspective. It provides the potential to eliminate spanning tree, first hop redundancy protocol needs, along with multiple touch points to configure those technologies. Using Multi-chassis EtherChannel (MEC), bandwidth can be effectively doubled with minimized convergence timers using stateful and graceful recovery. In traditional networks, StackWise virtual is positioned in the distribution layer and in collapsed core environments to help VLANs span multiple access layer switches, to provide flexibility for applications and services requiring Layer 2 adjacency, and to provide Layer 2 redundancy.

Layer 3 routed access moves the Layer 2/Layer 3 boundary from the distribution layer to the access layer. The distribution and collapsed core layers are no longer required to service the Layer 2 adjacency and Layer 2 redundancy needs with the boundary shifted. While StackWise Virtual can provide an operational simplicity for control plane protocols and physical adjacencies, it is at the expense of additional protocols designed to solve Layer 2 challenges, and, when leveraged in a Layer 3 routed network, can result in the loss of a redundant IGP/EGP control plane instance. In a Layer 3 routed access environment, two separate, physical switches are best used in all situations except those that may require Layer 2 redundancy.

For example, at the access layer, if physical hardware stacking is not available in the deployed platform, StackWise Virtual can be used to provide Layer 2 redundancy to the downstream endpoints. In LISP/VXLAN fabric, StackWise Virtual is best positioned in two places:

  • Edge Node—Downstream servers hosting virtual endpoints often require Layer 2 high availability. StackWise Virtual can provide multiple, redundant 1- and 10-Gigabit Ethernet connections common on downstream devices.

  • Fabric in a Box—When deploying a Fabric in a Box, if the given platform does not support hardware stacking, StackWise Virtual can provide redundancy and high availability.


Overlay Network Design

In the LISP/VXLAN fabric, the overlay networks are used for transporting user traffic across the fabric. The fabric encapsulation also carries Security Group information used for traffic segmentation inside the overlay VNs.

Consider the following in the design when deploying virtual networks:

  • Virtual Networks (Macro-segmentation)—Use virtual networks when requirements dictate isolation at both the data plane and control plane. In general, if devices need to communicate with each other, they should be placed in the same virtual network. If communication is required between different virtual networks, use an external firewall or other device to enable inter-VN communication. Virtual Network provides the same behavior and isolation as VRFs.

  • SGTs (Micro-segmentation)—Segmentation using SGTs allows for simple-to-manage Security Group policies and enables granular data plane isolation between groups of endpoints within a virtualized network. Using SGTs also enables scalable deployment of policy without having to do cumbersome updates for these policies based on IP addresses.

  • Reduce subnets and simplify DHCP management—In the overlay, IP subnets can be stretched across the fabric without flooding issues that can happen on large Layer 2 networks. Use fewer subnets and DHCP scopes for simpler IP addressing and DHCP scope management. Subnets are sized according to the services that they support, versus being constrained by the location of a gateway. Enabling the optional broadcast flooding (Layer 2 flooding) feature can limit the subnet size based on the additional bandwidth and endpoint processing requirements for the traffic mix within a specific deployment.

  • Avoid overlapping IP subnets—Different overlay networks can support overlapping address space but be aware that most deployments require shared services across all VNs, and some may use inter-VN communication. Avoid overlapping address space so that the additional operational complexity of adding a network address translation (NAT) device is not required for shared services communication.


Shared Services Design

This section is organized into the following subsections:

Section Subsection
Shared Services Design

Services Block Design

Shared Services Routing Table


Services Block Design

As campus network designs utilize more application-based services, migrate to controller-based WLAN environments, and continue to integrate more sophisticated Unified Communications, it is essential to integrate these services into the campus smoothly while providing for the appropriate degree of operational change management and fault isolation. And this must be done while continuing to maintain a flexible and scalable design.

A services block provides for this through the centralization of servers and services for the Enterprise Campus. The services block serves a central purpose in the campus design: it isolates or separates specific functions into dedicated services switches allowing for cleaner operational processes and configuration management. It also provides a centralized location for applying network security services and policies such as NAC, IPS, or firewall.

The services block is not necessarily a single entity. There might be multiple services blocks depending on the scale of the network, the level of geographic redundancy required, and other operational and physical factors. One services block may service an entire deployment, or each area, building, or site may have its own block.

The services block does not just mean putting more boxes in the network. It is the purpose-built linkage between the campus network and the end user services such as DHCP, DNS, Active Directory (AD), servers, and critical systems and the endpoint services such as the WLC and Unified Communication Systems.

Services blocks are delineated by the services block switch. The services block switch can be a single switch, multiple switches using physical hardware stacking, or be a multi-box, single logical entity such as StackWise Virtual (SVL), Virtual Switching System (VSS), or Nexus Virtual Port-Channels (vPCs). The goal of the services block switch is to provide Layer 3 access to the remainder of the enterprise network and Layer 2 redundancy for the servers, controllers, and applications in the services block. This allows the services block to keep its VLANs distinct from the remainder of the network stack such as the access layer switches which will have different VLANs.

WLCs, Unified Communication Services, and other compute resources should be interconnected with the service block switch using link aggregation (LAG). These Ethernet connections should be distributed among different modular line cards or switch stack members as much as possible to ensure that the failure of a single line card or switch does not result in total failure of the services to remainder of the network. Terminating on different modules within a single Catalyst and Nexus modular switch or different switch stack members provides redundancy and ensures that connectivity between the services block switch and the service block resources are maintained in the rare event of a failure.

The key advantage of using link aggregation is design performance, reliability, and simplicity. With the Ethernet bundle comprising up to eight links, link aggregation provides very high traffic bandwidth between the controller, servers, applications, and the remainder of the network. If any of the individual ports fail, traffic is automatically migrated to one of the other ports. If at least one port is functioning, the system continues to operate, remain connected to the network, and can continue to send and receive data.

When connecting wireless controllers to the services block using link aggregation, one of three approaches can be used:

  • Option 1—The WLCs are connected to the services block with a Layer 2 port-channel on each WLC connecting to each upstream switch. The links are spread across the physical switches. This is the recommended option.

  • Option 2—The WLCs are connected to the services block with a Layer 2 port-channel on each WLC without spreading the links across the physical switches. This is a variation of first option and is recommended only if the existing physical wiring will not allow for Option 1.

  • Option 3—If the services block is not operating in a logical configuration such as VSS, SVL, vPC, or a switch stack, then the first hop redundancy protocol (FHRP) HSRP should be used between the two devices in the services block. One WLC is connected via a port-channel trunk to the HSRP Active switch, and the other WLC is connected via a port-channel trunk to the HSRP Standby switch.

Figure 12. WLC High Availability Pair Connection Options

Slide18.JPG

 

 

Tech tip

For additional information regarding RP design and RP connectivity on the Catalyst 9800s WLC, please see: High Availability SSO Deployment Guide for Cisco Catalyst 9800 Series Wireless Controllers, Cisco IOS XE

Enterprise Campus deployments may span a large geographic area and be separated by MAN, WAN, or even public Internet circuits. If the survivability requirements for these locations necessitate network access, connectivity, and services in the event of egress circuit failure or unavailability, then a services block should be deployed at each physical location with these requirements. Commonly, medium to large deployments will utilize their own services block for survivability, and smaller locations will use centralized, rather than local services.

In very small sites, small branches, and remote sites, services are commonly deployed and subsequently accessed from a central location, generally a headquarters (HQ). However, due to the latency requirements for Fabric APs which operate in fabric mode, WLCs generally need to be deployed at each location. A maximum round trip time (RTT) of 20ms is required between a fabric mode access point and the WLC. For these very small or branch locations, a services block may not be needed if the only local service is the wireless LAN controller. Some deployments may be able to take advantage of either virtual or switch-embedded Catalyst 9800 WLC as discussed in the Embedded Wireless section.

Tech tip

If additional services are deployed locally such as an ISE PSN, AD, DHCP, or other compute resources, a services block will provide flexibility and scale while providing the necessary Layer 2 adjacency and high availability. A services block is the


Shared Services Routing Table

Once the services block physical design is determined, its logical design should be considered next. Shared services are commonly deployed in the global routing table (GRT) though they are also supported in a VRF. If deployed in a VRF, this routing table should be dedicated only to these shared services.

Discussed in detail later in the External Connectivity section, the endpoint prefix-space in the fabric site will be present on the border nodes for advertisement to the external world. However, these prefixes will be in a VRF table, not the global routing table. This later section discussion options on connecting the border node to shared services, Internet, and outside the fabric.

With shared services in a dedicated VRF, route leaking (VRF to VRF leaking) is administratively straightforward as it uses route-targets under the VRF configuration, although it is at the expense of creating another VRF to manage. The alternative approach, shared services in the GRT, requires a different approach to leak routes for access to shared services. The process still requires the same handoff components to the external entity to the border node, though with slightly more touch points. These begin with IP prefix-list for each VN in the fabric that references each of the associated subnets. A route-map is created to match on each prefix-list. Finally, the VRF configuration imports and exports routes that are filtered based on these route-maps.

While the second approach, shared services in GRT, may have more configuration elements, it also provides the highest degree of granularity. Specific routes can be selectively and systematically leaked from the global routing table to the fabric VNs without having to maintain a dedicated VRF for shared services. Both approaches are supported, although the underlying decision for the routing table used by shared services should be based on the entire network, not just the LISP/VXLAN fabric sites.


Fabric DHCP Overview and Design

LISP/VXLAN fabric does not require any specific changes to existing infrastructure services, because the fabric nodes have capabilities to handle the DHCP relay functionality differences that are present in fabric deployments. In a typical DHCP relay design, the unique gateway IP address determines the subnet address assignment for an endpoint in addition to the location to which the DHCP server should direct the offered address. In a fabric overlay network, that gateway is not unique—the same Anycast IP address exists across all fabric edge nodes within the fabric site. Without special handling either at the fabric nodes or by the DHCP server itself, the DHCP offer returning from the server may not be relayed to the correct edge node where the DHCP request originated.

Figure 13. Fabric DHCP Packet Flow

Slide19.JPG

 

 

Fabric DHCP Packet Flow

For simplicity, the DHCP Discover and Request packets are referred to as a DHCP REQUEST, and the DHCP Offer and Acknowledgement (ACK) are referred to as the DHCP REPLY.

  • Step 1—Endpoint sends a DHCP REQUEST to the edge node.

  • Step 2—The packet is inspected by DHCP Snooping.

  • Step 3a—Option 82 data (DHCP Relay Agent Information) is inserted into the DHCP REQUEST.

  • Step 3b—The Gateway IP address (giaddr) is set to the edge node’s Anycast IPv4 address (example: 172.16.201.1).

  • Step 4—Packet is encapsulated and sent to the border node where it is relayed to the DHCP server.

  • Step 5a—DHCP server receives the DHCP REQUEST and offers an IP address within the applicable scope. The original Option 82 information is echoed back in the DHCP REPLY.

  • Step 5b—DHCP server uses the Gateway IP address (giaddr) from DHCP REQUEST packet as the destination.

  • Step 6—The DHCP REPLY sent back toward the border, as it also has the same Anycast IPv4 address assigned to a Loopback interface.

  • Step 7—The DHCP REPLY is inspected, and the border node uses the option 82 information to determine the source RLOC (example: 192.168.255.2).

  • Step 8—DHCP REPLY packet is encapsulated and sent back to the original source edge node.

  • Step 9—Edge node receives the DHCP REPLY, de-encapsulates, and forwards to the endpoint which is identified via its MAC address.

To identify the specific DHCP relay source, the configuration of the Relay Agent at the fabric edge with DHCP option 82 is configured. When a fabric edge node receives a DHCP Discovery message, it adds the DHCP Relay Agent Information using option 82 to the DHCP packet and forwards it across the overlay.

Relay Agent Information is a standards-based (RFC 3046) DHCP option. It is a container option which contains two parts (two sub-options):

  • Agent Circuit ID—Identifies the VLAN, the interface module, and interface port number.

  • Agent Remote ID—Identifies the LISP Instance-ID (the VN), the IP Protocol (IPv4 or IPv6), and the source RLOC.

The relay agent sets the gateway address (giaddr field of the DHCP packet) as the IP address of the SVI the DHCP packet was received on. Once the DHCP option 82 information is inserted into the original packet, it is encapsulated in fabric VXLAN and forwarded across the overlay to the fabric border node who then forwards the packet to the DHCP server. The DHCP server, by referring to the relay agent IP address (giaddr) in a DHCP Discover message, allocates an address to the DHCP client from the address pool scope.

The border node has advanced DHCP relay capabilities which allows DHCP server configuration to remain unchanged for scopes covering fabric endpoints. Border nodes inspect the DHCP offer returning from the DHCP server. The offer includes the RLOC (edge node’s loopback) from fabric edge switch which relayed the original DHCP request. The border node references the embedded option 82 information and directs the DHCP offer back to the correct fabric edge destination. This reply is encapsulated in Fabric VXLAN and sent across the overlay.

Tech tip

The DHCP server used in the deployment must conform the RFC standard and echo back the Option 82 information. Modern Microsoft Windows Servers such as 2012 R2 and beyond generally adhere to this standard. Other DHCP server providers such as Infoblox and BlueCat also adhered to this standard, though support may vary by release. Please check the applicable manufacture’s release notes and user guides for the DHCP server in used in the deployment.


Latency

Latency in the network is an important consideration for performance, and the RTT between Cisco ISE and any network device it manages must be taken into strict account. The RTT should be equal to or less than the number mentioned in the below picture to achieve optimal performance.

Figure 14. LISP/VXLAN Network Latency Requirements

Slide21.JPG

 

 

Device Role Design Principles

This section is organized into the following subsections:

Section Subsection
Device Role Design Principles

Edge Node

Control Plane Node

Border Node

Fabric in a Box

Roles and Capabilities

This section discusses design principles for specific LISP/VXLAN fabric device roles including edge nodes, control plane nodes, border nodes, Fabric in a Box. This section concludes with device platform role and capabilities discussion.


Edge Node Design

In LISP/VXLAN fabric, edge nodes represent the access layer in a two or three-tier hierarchy. The access layer is the edge of the campus. It is the place where end devices attach to the wired portion of the campus network. The edge nodes also represent the place where devices that extend the network connectivity out one more layer connect. These include devices such as IP phones, access points.

The access layer provides the intelligent demarcation between the network infrastructure and the devices that leverage that infrastructure. As such it provides a trust boundary for QoS, security, and policy. It is the first layer of defense in the network security architecture, and the first point of negotiation between end devices and the network infrastructure.

To meet network application and end-user demands, Cisco Catalyst switching platforms operating as a fabric edge node do not simply switch packets but provide intelligent services to various types of endpoints at the network edge. By building intelligence into these access layer switches, it allows them to operate more efficiently, optimally, and securely.

The edge node design is intended to address the network scalability and availability for the IT-managed voice, video, and wireless communication devices along with the wide variety of possible wired endpoint device types. Edge nodes should maintain a maximum 20:1 oversubscription ratio to the distribution or collapsed core layers. The higher the oversubscription ratio, the higher the probability that temporary or transient congestion of the uplink may occur if multiple devices transmit or receive simultaneously. Uplinks recommendation should be minimum of 10 Gigabit Ethernet and should be connected to multiple upstream peers.

As new devices are deployed with higher power requirements, such as lighting, surveillance cameras, virtual desktop terminals, remote access switches, and APs, the design should have the ability to support power over Ethernet to at least 60W per port, offered with Cisco Universal Power Over Ethernet (UPOE), and the access layer should also provide PoE perpetual power during switch upgrade and reboot events. New endpoints and building systems may require even more power, and IEEE 802.3bt and Cisco UPOE-Plus (UPOE+) can provide power up to 90W per port. Both fixed configuration and modular switches will need multiple power supplies to support 60–90W of power across all PoE-capable ports.


Control Plane Node Design

The fabric control plane node contains the database used to identify an endpoint’s location in the network. This is a central and critical function for the fabric to operate. A control plane node that is overloaded and slow to

respond results in application traffic loss on initial packets. If the fabric control plane is down, endpoints inside the fabric fail to establish communication to remote endpoints that are not cached in the local database.

For redundancy, it is recommended to deploy two control plane nodes to ensure high availability of the fabric site, as each node contains a copy of control plane information acting in an Active/Active state. The devices supporting the control plane should be chosen to support the HTDB (EID-to-RLOC bindings), CPU, and memory needs for an organization based on the number of endpoints. Border nodes and edge nodes register with and use all control plane nodes, so redundant nodes chosen should be of the same type for consistent performance.

The Cisco Catalyst Wireless LAN Controller (WLC) has the capability to communicate with a maximum of two Control Plane (CP) nodes within a fabric site. However, if the Multisite Remote Border feature (which will be discussed later in the guide) is implemented, then we can support up to 16 pairs of Control Plane (CP) nodes. (One CP pair is local to the site)


Distributed Control Plane Node and Border Node

The control plane node advertises the fabric site prefixes learned from the LISP protocol to certain fabric peers, I.e., the border nodes. When the control plane nodes are deployed as dedicated devices, not co-located with other fabric roles, they provide the highest degrees of performance, reliability, and availability.

Control plane nodes may be deployed as either dedicated (distributed) or non-dedicated (co-located) devices from the fabric border nodes. In a Fabric in a Box deployment, fabric roles must be co-located on the same device. In Small and Very Small deployment, as discussed in the Reference Models section below, it is not uncommon to deploy a co-located control plane node solution, utilizing the border node and control plane node on the same device. Deploying a dedicated control plane node has advantages in Medium and Large deployments as it can provide improved network stability both during fabric site change management and in the event that a fabric device becomes unavailable in the network.

Dedicated control plane nodes, or off-path control plane nodes, which are not in the data forwarding path, can be conceptualized using the similar DNS Server model. The control plane node is used for LISP control plane queries, although it is not in the direct data forwarding path between devices. The physical design result is similar to a Router on a Stick topology.

The dedicated control plane node should have ample available memory to store all the registered prefixes. Bandwidth is a key factor for communication prefixes to the border node, although throughput is not as key since the control plane nodes are not in the forwarding path. If the dedicated control plane node is in the data forwarding path, such as at the distribution layer of a three-tier hierarchy, throughput should be considered along with ensuring the node is capable of CPU-intensive registrations along with the other services and connectivity it is providing.

One other consideration for separating control plane functionality onto dedicated devices is to support frequent roaming of endpoints across fabric edge nodes. Roaming across fabric edge nodes causes control plane events in which the WLC updates the control plane nodes on the mobility (EID-to-RLOC mapping) of these roamed endpoints. Although co-located control plane is the simplest design, adding the control plane node function on border nodes in a high-frequency roam environments can lead to high CPU on co-located devices. For high-frequency roam environments, a dedicated control plane node should be used.


Co-located Control Plane Node and Border Node

If the chosen border nodes support the anticipated endpoint, throughput, and scale requirements for a fabric site, then the fabric control plane functionality can be co-located with the border node functionality. While it does provide operational simplicity in that it is two less pieces of equipment to manage, it also reduces the potential for resiliency in the event of software upgrade, device reboots, common upgrades, or updates to configuration.


Border Node Design

A border node is an entry and exit point to the fabric site. In effect, it speaks two languages: LISP/VXLAN fabric on one link and traditional routing and switching on another. The fabric border design is dependent on how the fabric site is connected to networks outside of the fabric site.

Border node functionality is supported on switching platforms. The correct platform should be selected for the desired outcome. A border node does not have a direct mapping to a layer in the network hierarchy. However, the border node is not necessarily a distribution layer switch or core switch in the network.


Border Nodes and External Networks

When configuring a border node in LISP/VXLAN fabric, there are three different options to indicate the type of external network(s) to which the device is connected.

A border may be connected to internal, or known, networks such as data center, shared services, and private WAN. Routes that are learned from the data center domain are registered with the control plane node, similarly to how an edge node registers an endpoint. In this way, LISP, rather than native routing, is used to direct traffic to these destinations outside of the fabric.

In Figure 15 below, there are two sets of border nodes. The external border nodes connect to the Internet and to the rest of the Campus network. The internal border nodes connect to the Data Center by way of VRF-Aware peers (fusion/peer devices). If traditional, default forwarding logic is used to reach the Data Center prefixes, the fabric edge nodes would send the traffic to the external border nodes who would then hairpin the traffic to the internal border nodes resulting in an inefficient traffic forwarding. By importing, or registering, the Data Center prefixes with the control plane node using the internal border functionality, edge nodes can send traffic destined for 198.18.133.0/24 directly to the internal border nodes. Traffic destined for the Internet and remainder of the campus network to the external border nodes.

Figure 15. Internal Border Node Example

Slide22.JPG

 

 

A border may be connected to external, or unknown, networks such as Internet, WAN, or MAN. The routes learned from the external domain are not registered (imported) to the control plane node. This border is the default exit point, or gateway of last resort, for the virtual networks in the fabric site.

The control plane node has a mechanism that notifies the fabric devices that a destination prefix is not registered with it. This triggers the device requesting this mapping to simply send traffic to the external border node. Most LISP/VXLAN fabric deployments should provision border nodes as external which provisions the device as the fabric site gateway of last resort. This deployment type uses default routing (traditional forwarding logic), rather than LISP, to reach all external prefixes.

In Figure 16 below, there are a single pair of borders nodes that represent the common egress point from the fabric site. The border nodes are connected to the Data Center, to the remainder of the campus network, and to the Internet. When the edge nodes forward traffic to any of these external destinations, the same border nodes will be used. Traditional, default forwarding logic can be used to reach these prefixes, and it is not necessary to register the Data Center prefixes with the control plane node.

Figure 16. External Border Node Example

Slide23.JPG

 

 

A border node may also be connected to both known and unknown networks such as being a common egress point for the rest of an enterprise network along with the Internet. What distinguishes this border is that known routes such as shared services and data center, are registered with the control plane node rather than using the default forwarding logic described above. This type of border node is sometimes referred to as an Anywhere border node.

The key distinction between these border types is the underlying routing logic that is used to reach known prefixes. Networks deployed similarly to Figure 8 – LISP/VXLAN Fabric Roles (Example) do not commonly import (register) routes with the control plane node. Because there is a common egress point to the fabric site, the

border nodes are the destination for both known and unknown external routes. Registering the known external prefixes in this type of design is not needed, as the same forwarding result is achieved for both known and unknown prefixes. Most deployments should provision a border node using the external border node type.

In Figure 17 below, both border nodes are connected to the Internet and to the remainder of the campus network. Each border node is also connected to a separate Data Center with different prefixes. If traditional, default forwarding logic is used to reach these prefixes, the fabric edge nodes may send the traffic to a border not directly connect to the applicable data center. Traffic will have to inefficiently traverse the crosslink between border nodes. By importing the data center prefixes into LISP, the edge nodes can send to the traffic to the border node on the left to reach 203.0.113.0/24 and the border node on the right to reach 198.51.100.0/24. Either border can be used as the default path to the Internet.

Figure 17. Anywhere Border Node Example

Slide24.JPG

 

 

Fabric in a Box Design

Some physical locations may use unique wiring plans such that the MDF and IDF do not conform to the common two-tier and three-tier hierarchical network structure. The result is that the available fiber and copper wiring may require access switches to be daisy-chained or configured in a ring. Any number of wiring variations may exist in a deployment. Due to the unique nature of supporting all three fabric roles on a node, Fabric in a Box has specific topologies that are supported if additional fabric edge nodes are connected to it (downstream from it). The topologies supported differ based on if LISP/VXLAN fabric Embedded wireless is also implemented.

In locations where physical stacking is not possible due to the wiring structure, Fabric in a Box can support up to two daisy-chained edge nodes creating a three-tier topology. Embedded wireless is also supported in this scenario. Dual Fabric in a Box is also supported, though should only be used if mandated by the existing wiring structures. Fabric in a Box is supported using a single switch, a switch with hardware stacking, or with StackWise Virtual deployment.


Platform Roles and Capabilities Considerations

The LISP/VXLAN fabric network platform should be chosen based on the capacity and capabilities required by the network, considering the recommended functional roles.

  • Only Cisco Catalyst 9000 Series switches are supported with LISP/VXLAN fabric. Refer to Fabric Roles Supported by Cisco Catalyst 9000 Series Switches for exact details.

  • Cisco Catalyst 9800 Series Wireless Controller that is available in multiple form factors such as an Appliance, Cloud-based, or Embedded Wireless for a Switch.

  • Wi-Fi 6 Access Points, which are the Cisco Catalyst 9100 Series APs.

  • 802.11ac Wave 2 Access Points, which are the AP1540 Series, AP1560 Series, AP1800 Series, AP2800 Series, AP3800 Series, and AP4800 Series.


Feature-Specific Designs

This section is organized into the following subsections:

Section Subsection
Feature-Specific Design Requirements

Multicast

Layer 2 Flooding

Critical VLAN

The following section discusses design consideration for specific features in LISP/VXLAN fabric. It begins with a discussion on multicast design, traditional multicast operations, and Rendezvous Point design and placement. Multicast forwarding in the fabric is discussed along with considerations regarding the Layer 2 flooding feature which relies on a multicast transport in the underlay. Next, Critical VLAN is described along with considerations for how it is deployed in LISP/VXLAN fabric.


Fabric Multicast Overview

Multicast is supported both in the overlay virtual networks and the in the physical underlay networks in SD- Access, with each achieving different purposes as discussed further below.

The multicast source can either be outside the fabric site (commonly in the data center) or can be in the fabric overlay, directly connected to an edge node or associated with fabric AP. Multicast receivers are commonly directly connected to edge nodes although can also be outside of the fabric site if the source is in the overlay.

PIM Any-Source Multicast (PIM-ASM) and PIM Source-Specific Multicast (PIM-SSM) are supported in both the overlay and underlay. The overlay or the underlay can be used as the transport for multicast as described in the Forwarding section.


Rendezvous Point Design

In PIM-ASM routing architecture, the multicast distribution tree is rooted at the Rendezvous Point (RP). This is referred to as shared tree or RP-Tree (RPT), as the RP acts as the meeting point for sources and receivers of multicast data. The advantage of using RPs is that multicast receivers do not need to know about every possible source, in advance, for every multicast group. Only the address of the RP, along with enabling PIM, is needed to begin receiving multicast streams from active sources.

A Rendezvous Point is a switch (a Layer-3 device) in a multicast network that acts as a shared root for the multicast tree. Rendezvous Points can be configured to cover different multicast groups, or with regards to LISP/VXLAN fabric, cover different virtual networks. Active multicast sources are registered with an RP, and network devices with interested multicast receivers will join the multicast distribution tree at the Rendezvous Point.

An RP can be active for multiple multicast groups, or multiple RPs can be deployed to each cover individual groups. The information on which RP is handling which group must be known by all the switches in the multicast domain. For this group-to-RP-mapping to occur, multicast infrastructure devices must be able to locate the Rendezvous Point in the network. In traditional multicast networks, this can be accomplished through static RPs, BSR (Boot Strap Router), Auto-RP, or Anycast-RP.

Anycast-RP allows two or more RPs to share the load for multicast source registration and act as hot-standbys for each other. With multiple, independent RPs in the network, a multicast source may register with one RP and a receiver may register with another, as registration is done with the closest RP (in terms of the IGP metric).

Anycast-RP uses MSDP (Multicast Source Discovery Protocol) to exchange source-active (SA) information between redundant RPs. This allows the sources to be known to all the Rendezvous Points, independent of which one received the multicast source registration. Anycast-RP is the preferred method in LISP/VXLAN fabric.


Rendezvous Point Placement

Where an RP is placed in a network does not have to be a complex decision. To aid in this decision process, it can be helpful to compare PIM-ASM and PIM-SSM and understand the multicast tree building.

Protocol independent multicast (PIM) is used to build a path backwards from the receiver to the source, effectively building a tree. This tree has a root with branches leading out to the interested subscribers for a given stream. With PIM-ASM, the root of the tree is the Rendezvous Point. With PIM-SSM, the root of the multicast tree is the source itself.

Source tree models (PIM-SSM) have the advantage of creating the optimal path between the source and the receiver without the need to meet a centralized point (the RP). In a shared tree model (PIM-ASM), the path through the RP may not be the shortest path from receiver back to source. However, PIM-ASM does have an automatic method called switchover to help with this. Switchover moves from the shared tree, which has a path to the source by way of the rendezvous point, to a source tree, which has a path directly to the source. This capability provides an automatic path optimization capability for applications that use PIM-ASM.

In an environment with fixed multicast sources, RPs can easily be placed to provide the shortest-path tree. In environments with dynamic multicast sources, RPs are commonly placed in the core of a network. In traditional networking, network cores are designed to interconnect all modules of the network together, providing IP reachability, and generally have the resources, capabilities, and scale to support being deployed as a Rendezvous Point.

In LISP/VXLAN fabric networks, border nodes act as convergence points between the fabric and non-fabric networks. Border nodes are effectively the core of the network. Discussed above, border node device selection is based on the resources, scale, and capability to support being this aggregation point between fabric and non-fabric.

Multicast sources are commonly located outside the fabric site such as with Music on Hold (MOH), streaming video/video conferencing, and live audio paging and alert notifications. For unicast and multicast traffic, the border nodes must be traversed to reach destinations outside of the fabric. The border nodes already represent the shortest path.

Most environments can achieve the balance between optimal RP placement along with having a device with appropriate resources and scale by selecting their border node as the location for their multicast Rendezvous Point.


External RP

The Rendezvous Point does not have to be deployed on a device within the fabric site. External devices can be designated as RPs for the multicast tree in a fabric site. The External RP address must be reachable in the VN routing table on the border nodes. External RP placement allows existing RPs in the network to be used with the fabric. In this way multicast can be enabled without the need for new MSDP connections. If RPs already exist in the network, using these external RPs is the preferred method to enable multicast. If Layer 2 flooding functionality is desired within a fabric, it would be good to restrict flooding within the fabric by having Border nodes as RPs.


Multicast Forwarding in LISP/VXLAN fabric.

LISP/VXLAN fabric supports two different transport methods for forwarding multicast. One uses the overlay and is referred to as head-end replication, and the other uses the underlay and is called Native Multicast. Multicast forwarding is enabled per-VN. However, if native-multicast is enabled, for a VN, head-end replication cannot be used for another VN in the fabric site. These two options are mutually exclusive within the fabric site.


Head-End Replication

Head-end replication (or ingress replication) is performed either by the multicast first-hop router (FHR), when the multicast source is in the fabric overlay, or by the border nodes, when the source is outside of the fabric site.

The multicast packets from the source are replicated and sent, via unicast, by the FHR to all last-hop routers (LHR) with interested subscribers.

For example, consider a fabric site that has twenty-six (26) edge nodes. Each edge node has receivers for a given multicast group, and the multicast source is connected to one of the edge nodes. The FHR edge node must replicate each multicast packet to all other twenty-five edge nodes. This replication is performed per source, and packets are sent across the overlay. A second source means another twenty-five unicast replications. If the multicast source is outside of the fabric site, the border node acts as the FHR for the fabric site and performs the head-end replication to all fabric devices with interested multicast subscribers.

Figure 18. LISP/VXLAN Fabric Multicast Head-End Replication Packet

image25.png

 

The advantage of head-end replication is that it does not require multicast in the underlay network. This creates a complete decoupling of the virtual and physical networks from a multicast perspective. However, this can create high overhead on the FHRs and result in high bandwidth and CPU utilization. In deployments where multicast cannot be enabled in the underlay networks, head-end replication can be used. Networks should consider Native Multicast due to its efficiency and the reduction of load on the FHR fabric node. Head-End Replication can be used in scenarios where the network is a two-tier topology where the outgoing interfaces are many connecting to many switches in the access layer.


Native Multicast

Native multicast does not require the ingress fabric node to do unicast replication. Rather the whole underlay, including intermediate nodes (nodes not operating in a fabric role) are used to do the replication. To support native multicast, the FHRs, LHRs, and all network infrastructure between them must be enabled for multicast.

Native multicast uses PIM-SSM for the underlay multicast transport. The overlay multicast messages are tunneled inside underlay multicast messages. This behavior also allows overlap in the overlay and underlay multicast groups in the network, if needed. Because the entire underlay network between source and receiver is working to do the packet replication, scale and performance is vastly improved over head-end replication.

Native multicast works by performing multicast-in-multicast encapsulation. Multicast packets from the overlay are encapsulated in multicast in the underlay. With this behavior, both PIM-SSM and PIM-ASM can be used in the overlay. Native multicast can be used in scenarios where the network is a three-tier topology and there are a couple of outgoing interfaces to downstream switches aggregating the connections from the access layer.

Figure 19. LISP/VXLAN Fabric Native Multicast Packet

image26.png

 

Layer 2 Flooding

Layer 2 flooding is feature that enables the flooding of broadcast, link-local multicast, and ARP traffic for a given overlay subnet. In traditional networking, broadcasts are flooded out of all ports in the same VLAN. By default, LISP/VXLAN fabric transports frames without flooding Layer 2 broadcast and unknown unicast traffic, and other methods are used to address ARP requirements and ensure standard IP communication gets from one endpoint to another.

However, some networks need to utilize broadcast, particularly to support silent hosts which generally require reception of an ARP broadcast to come out of silence. This is commonly seen in some building management systems (BMS) that have endpoints that need to be able to ARP for one other and receive a direct response at

Layer 2. Another common use case for broadcast frames is Wake on LAN (WoL) Ethernet broadcasts which occur when the source and destination are in the same subnet.

Because the default behavior, suppression of broadcast, allows for the use of larger IP address pools, pool size of the overlay subnet needs careful consideration when Layer 2 flooding is enabled. Consider using a /24 (24- bit netmask) or smaller address pool to limit the number of broadcasts, as each of these frames must be processed by every device in the segment. Layer 2 flooding should be used selectively, where needed, using small address pool.

Layer 2 flooding works by mapping the overlay subnet to a dedicated multicast group in the underlay. Broadcast, link-local multicast, and ARP traffic are encapsulated in fabric VXLAN and sent to the destination underlay multicast group. PIM ASM is used as the transport mechanism. It is recommended that the RPs for this underlay multicast ASM group be configured on the intermediate switches or on the Border nodes of the fabric. It is not necessary to forward this ASM group to the normal RP used for data traffic. MSDP and BGP can be used to form an Anycast-RP.

When Layer 2 flooding is enabled for a given subnet, all edge nodes will send multicast PIM joins for the respective underlay multicast group, effectively pre-building a multicast shared tree. A shared tree must be rooted at a Rendezvous Point, and for Layer 2 flooding to work, this RP must be in the underlay.

For Layer 2 flooding to work, multicast routing needs to be configured on the devices in the fabric site and MSDP should be configured between the RPs in the underlay. Loopback 0 can be used as the connect-source and originator-ID for the MSDP peering.

Connect-source uses the primary IP address on the configured interface as the source IP address of the MSDP TCP connection. Originator-ID allows the MSDP speaker originating a source-active (SA) message to use the IP address of the defined interface as the RP address of the message. Originator-ID is the inherent mechanism by which MSDP works to address the RPF check.


About Critical VLAN

By default, when a network access device (NAD) cannot reach its configured RADIUS servers, new hosts connected to the NAD cannot be authenticated and are not provided access to the network. The inaccessible authentication bypass feature, also referred to as critical authentication, AAA fail policy, or simply critical VLAN, allows network access on a particular VLAN when the RADIUS server is not available (down).

When a NAD tries to authenticate an endpoint connected to a port, it first checks the status of the configured RADIUS servers. If a server is available, the NAD can authenticate the host. If all the configured RADIUS servers are unavailable and the critical VLAN feature is enabled, the NAD grants network access to the endpoint and puts the port in the critical-authentication state which is a special-case authentication state. When the RADIUS servers are available again, clients in the critical-authentication state must reauthenticate to the network.

Similarly, critical voice VLAN support works by putting voice traffic into the configured voice VLAN if the RADIUS server becomes unreachable.


Critical VLAN Design Considerations

Within a fabric site, a subnet can be assigned to the critical data VLAN. The critical voice VLAN does not need to be explicitly defined, as the same VLAN is used for both voice and critical voice VLAN support. This ensures that phones will have network access whether the RADIUS server is available or not.

As discussed in the Fabric Overlay Design section, LISP/VXLAN fabric creates segmentation in the network using two methods: VRFs (Virtual networks) for macro-segmentation and SGTs (Security Group Access Control) for micro- segmentation. By default, users, devices, and applications in the same VN can communicate with each other. SGTs can permit or deny this communication within a given VN.

When designing the network for the critical VLAN, this default macro-segmentation behavior must be considered. For example, consider if the subnet assigned for development servers is also defined as the critical VLAN. In the event of the RADIUS server being unavailable, new devices connecting to the network will be placed in the same VLAN as the development servers. Because these devices are in the same VN, communication can occur between them. This is potentially highly undesirable.

Creating a dedicated VN with limited network access for the critical VLAN is the recommended and most secure approach. In the event of RADIUS unavailability, new devices connecting to the network will be placed in their own virtual network which automatically segments their traffic from any other, previously authenticated hosts.

The dedicated critical VN approach must look at the lowest common denominator with respect to total number of VN supported by a fabric device. Certain switch models support only one or four users defined VNs. Using a dedicated virtual network for the critical VLAN may exceed this scale depending on the total number of other user defined VNs at the fabric site and the platforms used.


Wireless Design

This section is organized into the following subsections:

Section Subsection
Wireless Design

Over-the-Top Wireless

Guest Wireless

LISP/VXLAN fabric supports two options for integrating wireless access into the network. One option is to use traditional Cisco Unified Wireless Network (CUWN) local-mode configurations over-the-top as a non-native service. In this mode, the LISP/VXLAN fabric is simply a transport network for the wireless traffic, which can be useful during migrations to transport CAPWAP-tunneled endpoint traffic from the APs to the WLCs. The other option is fully integrated LISP/VXLAN fabric Wireless, extending the LISP/VXLAN fabric beyond wired endpoints to also include wireless endpoints.

Integrating the wireless LAN into the fabric provides the same advantages for the wireless clients as provided to the wired clients in the fabric, including addressing simplification, mobility with stretched subnets, and end-to- end segmentation with policy consistency across the wired and wireless domains. Wireless integration also enables the WLC to shed data plane forwarding duties while continuing to function as the control plane for the wireless domain.

Fabric wireless controllers manage and control the fabric-mode APs using the same general model as the traditional local-mode controllers which offers the same operational advantages such as mobility control and radio resource management. A significant difference is that client traffic from wireless endpoints is not tunneled from the APs to the wireless controller. Instead, communication from wireless clients is encapsulated in VXLAN by the fabric APs which build a tunnel to their first-hop fabric edge node. Wireless traffic it tunneled to the edge nodes as the edge nodes provide fabric services such as the Layer 3 Anycast Gateway, policy, and traffic enforcement.

This difference enables a distributed data plane with integrated SGT capabilities. Traffic forwarding takes the optimum path through the LISP/VXLAN fabric to the destination while keeping consistent policy, regardless of wired or wireless endpoint connectivity.

The control plane communication for the APs does use a CAPWAP tunnel to the WLC, which is like the traditional CUWN control plane. However, a fabric WLC is integrated into the LISP/VXLAN fabric control plane (LISP) communication. When added as a Fabric WLC, the controller builds a two-way communication to the fabric control plane nodes.

Tech tip

Border nodes and edge nodes also build this two-way communication, or LISP session, with the control plane nodes.

This communication allows the WLCs to register client Layer 2 MAC addresses, SGT, and Layer 2 segmentation information (Layer 2 VNI). All of this works together to support wireless client roaming between APs across the fabric site. The LISP/VXLAN fabric control plane process inherently supports the roaming feature by updating its host-tracking database when an endpoint is associated with a new RLOC (wireless endpoint roams between APs).


Fabric Wireless Integration Design

Fabric-mode APs connect into a Default instance of LISP. This instance id is associated with the global routing table (GRT). This design allows the WLC to connect into the fabric site for AP management without needing to leak routes out of a VRF table.

When integrating fabric-enabled wireless into the LISP/VXLAN fabric architecture, the WLC control plane keeps many of the characteristics of a local-mode controller, including the requirement to have a low-latency connection between the WLC and the APs. This latency requirement, 20ms RTT, precludes a fabric WLC from managing fabric-mode APs at a remote site across a typical WAN. As a result, a remote site with LISP/VXLAN fabric wireless with a WAN circuit exceeding 20ms RTT will need a WLC local to that site.

Wireless integration with LISP/VXLAN fabric should also consider WLC placement and connectivity. WLCs typically connect to a shared services distribution block that is part of the underlay. The preferred services block has chassis redundancy as well as the capability to support Layer 2 multi-chassis EtherChannel connections for link and platform redundancy to the WLCs. As described in the Services Block section, VSS, StackWise Virtual, switch stacks, and Nexus vPC can be used to accomplish these goals.

In the simplified example diagram below, the border nodes are directly connected to the services block switch with Layer 3 connections. The WLCs are connected to the services block using link aggregation. Each WLC is connected to member switch of the services block logical pair.

Figure 20. Simplified WLC, Services Block, and Border Node Topology

Slide26.JPG

 

 

 

 

Over-the-Top Centralized Wireless Design

In cases where the WLCs and APs cannot participate in the fabric, a traditional CUWN centralized design model is an option. In Centralized WLC deployment models, WLCs are placed at a central location in the enterprise network. With this deployment model, the CAPWAP tunnels between WLC and APs traverse the campus backbone network. In the over-the-top model, this means the wireless infrastructure uses the fabric as a transport but without the benefits of fabric integration.

An over-the-top wireless design still provides AP management, simplified configuration, and troubleshooting, and roaming at scale. In this centralized over-the-top model, the WLAN controller is connected at the data center services block or a dedicated service block adjacent to the campus core. Wireless traffic between WLAN clients and the LAN is tunneled using CAPWAP between APs and the controller. APs can reside inside or

outside the fabric without changing the centralized WLAN design. However, the benefits of fabric and LISP/VXLAN fabric are not extended to wireless when it is deployed over-the-top.


Guest Wireless Design

When designing for Guest Wireless, LISP/VXLAN fabric supports two different models:

  • Guest as a dedicated VN—Guest is simply another user-defined VN.

  • MultiSite Remote Border— Multisite Remote Border allows the same IP address pool to exist at multiple fabric sites through Anchoring the pool to specific border node(s) and control plane node(s).


Guest as a VN

With Guest as VN, guest and enterprise clients share the same control plane node and border node. The Guest SSID is associated to a dedicated Guest VN, and SGTs are used for isolating guest traffic from itself. Guests, by the nature of VRFs and macro segmentation, are automatically isolated from other traffic in different VNs though the same fabric nodes are shared for guest and non-guest.

Guest users should be assigned an SGT value upon connecting to the network. This assignment is used to implement an equivalence of a peer-to-peer blocking policy. For a Fabric SSID, all security policy is enforced at the edge node, not at the access point itself. Traditional peer-to-peer blocking, which is enabled on the WLAN in the WLC, would not take effect. To provide consistent policy, an AP will forward traffic to the fabric edge, even if the clients communicating are associated with the same AP. An SGT assigned to Guest users can be leveraged to deny traffic between the same SGTs.

When designing for Guest as a VN, the same design modalities referenced throughout this document for any other virtual network apply to this Guest VN.


Multisite Remote Border

This design leverages a dedicated control plane node and border node for guest traffic. The nodes can be co-located on the same device, for operational simplicity, or on separate devices, for maximum scale and resilience. This functionality provides a simplified way to tunnel the Guest traffic to the DMZ which is a common security convention.

Figure 21. Multisite Remote Border

Slide32.JPG

 

 

 

This feature enables multiple fabric sites to use one common external border node. This capability enables a virtual network to be available across multiple sites, and the same subnet that is associated with the virtual network is spread across sites. A common external border and control plane is assigned to terminate all traffic in this virtual network, regardless of where the traffic originates.

Like the enterprise traffic, guest traffic is still encapsulated in VXLAN at the AP and sent to the edge node. The edge node is configured to use the anchor border node and control plane node as well as the enterprise nodes. All guest traffic is encapsulated in fabric VXLAN by the edge node and tunneled to the anchored border node. The anchored border node commonly resides in the DMZ to provide complete isolation from the

enterprise traffic. Guest users are registered to an anchored control plane node in DMZ.

Any VN (not just Guest) can be anchored if egress traffic needs to exit the network at a specific location (site).

  • The Anchored VN is first deployed at one fabric site (Anchor Site).

  • The Anchor Site defines the border and control plane node for the Anchored VN.

  • Anchored VN exists at other fabric sites (Anchoring Sites) and sends traffic to the Anchor Site.

  • VXLAN cannot be fragmented (DF-bit set); MTU must be sufficient between Anchor Site and Anchoring Sites.

  • Each fabric node hosting an Anchor VN must have a /32 RIB entry for all other fabric nodes hosting the same Anchor VN.

  • Seamless wireless roaming between fabric sites is not supported.


External Connectivity

This section is organized into the following subsections:

Section Subsection
External Connectivity

Layer 3 Handoff

VRF-Aware Peer

Non-VRF-Aware Peer

Firewall Peer

External Connectivity Considerations

External connectivity outside of the fabric site can have several possible variations, and these variations are based on underlying network design. For example, the fabric border node may be connected to an actual Internet edge router, an ISP device, a firewall, a services block switch, or some other routing infrastructure device. Each of these peer devices may be configured with a VRF-aware connection (VRF-lite) or may simply connect to the border node using the global routing table.

Shared services, as discussed in the earlier Routing Table section, may be deployed in a dedicated VRF or the global routing table, and shared services may be connected to a services block or be accessed through data center infrastructure. Internet access itself may be in a VRF, though is most commonly available in the global routing table. While each of these options are viable, though each present a different underlying network design that the fabric site must integrate with.


Layer 3 Handoff

Regardless of the potential variations for the network design and deployment outside of the fabric site, a few things are going to be in common, and the border node will be the device tying these things together:

  • VRF Aware—A border node will be VRF-aware. All user defined VNs in the fabric site are instantiated and provisioned as VRFs.

  • Site Prefixes in VRF—The EID-space prefixes associated with the fabric site will be in VRF routing tables on the border node.

  • Upstream Infrastructure—The border nodes will be connected to a next-hop device and further routing infrastructure (referenced simply as next-hop, for brevity).

Configuration on the border nodes to upstream infrastructure is done through an IP- based Layer 3 handoff. By IP-based, this means native IP forwarding, rather than encapsulation, is used. The fabric packet is de-encapsulated before being forwarded. The configuration is Layer 3 which means it uses Switched Virtual Interfaces (SVIs), as the border node is a switching platform, to connect to the upstream peers.

This Layer 3 handoff configuration provisions VRF-lite by associating each SVI with a different fabric VN (VRF). External BGP is used as the routing protocol to advertise the endpoint space (EID-space) prefixes from the fabric site to the external routing domain and to attract traffic back to the EID-space. This BGP peering can also be used to advertise routes into the overlay such as for access to shared services.

With the Layer 3 IP-based handoff configured, there are several common configuration options for the next-hop.

device. This device may peer (have IP connectivity and routing adjacency) with the border node using VRFs. This next-hop device may even continue the VRF segmentation extension to its next hop. This next-hop may not be VRF-aware and peer to the border node using the global routing table. Finally, the next-hop may be firewall which is special case peering that is not VRF-aware.


VRF-Aware Peer Design

This VRF-Aware peer design begins with VRF-lite configuration on the borer node and the peer configured as VRF-aware. For each VN that is handed off on the border node, a corresponding VN and interface is configured on the peer device. Existing collateral may refer to this deployment option as a fusion router or simply fusion device which is nothing but a peer device.

The generic term fusion router comes from MPLS Layer 3 VPN. The basic concept is that the fusion router is aware of the prefixes available inside each VPN (VRF), generally through dynamic routing, and can therefore fuse these routes together. In MPLS Layer 3 VPN, these generic fusion routers are used to route traffic between separate VRFs (VRF leaking). Alternatively, the fusion router can also be used to route traffic to and from a VRF to a shared pool of resources in the global routing table (route leaking). Both responsibilities are essentially the same as they involve advertising routes from one routing table into a separate routing table.

This VRF-Aware peer design is commonly used for access to shared services. Shared services are generally deployed using a services block deployed on a switching platform to allow for redundant and highly available Layer 2 links to the various devices and servers hosting these services. Shared service most commonly exists in the global routing table, though deployments may use a dedicated VRF to simply configuration.

In a LISP/VXLAN fabric deployment, the fusion device has a single responsibility: to provide access to shared services for the endpoints in the fabric. There are two primary ways to accomplish this task depending on how the shared services are deployed, route leaking and VRF leaking. Both require the fusion device to be deployed as VRF-aware.

  • Route Leaking—The option is used when the shared services routes are in the GRT. On the fusion device, IP prefix lists are used to match the shared services routes, route-maps reference the IP prefix lists, and the VRF configurations reference the route-maps to ensure only the specifically matched routes are leaked.
  • VRF Leaking—The option is used when shared services are deployed in a dedicated VRF on the fusion device. Route-targets under the VRF configuration are used to leak between the fabric VNs and the shared services VRF.

A fusion device can be either a true routing platform, a Layer 3 switching platform, or a firewall must meet several technological requirements. It must support:

  • Multiple VRFs—Multiple VRFs are needed for the VRF-Aware peer model. For each VN that is handed off on the border node, a corresponding VN and interface is configured on the peer device. The selected platform should support the number of VNs used in the fabric site that will require access to shared services.

  • Subinterfaces (Routers or Firewall)—A virtual Layer 3 interface that is associated with a VLAN ID on a routed physical interface. It extends IP routing capabilities to support VLAN configurations using the IEEE 802.1Q encapsulation.

  • Switched Virtual Interfaces (Layer 3 switch)—Represents a logical Layer 3 interface on a switch. This SVI is a Layer 3 interface forwarding for a Layer 3 IEEE 802.1Q VLAN.

  • IEEE 802.1Q—An internal tagging mechanism which inserts a 4-byte tag field in the original Ethernet frame between the Source Address and Type/Length fields. Devices that support SVIs and subinterfaces will also support 802.1Q tagging.

  • BGP-4—This is the current version of BGP and was defined in RFC 4271 (2006) with additional update RFCs. Along with BGP-4, the device should also support the Multiprotocol BGP Extensions such as AFI/SAFI and Extended Community Attributes defined in RFC 4760 (2007).

To support this route leaking responsibility, the device should be properly sized according to the number of VRFs, bandwidth and throughput requirements, and Layer 1 connectivity needs including port density and type. When the network has been designed with a services block, the services block switch can be used as the fusion device (VRF-aware peer) if it supports the criteria described above. Fusion devices should be deployed in pairs or as a multi-box, single logical box such as VSS, SVL, or vPC. When the fusion device is a logical unit, border nodes should be connected to both members of the logical pair as described in the later external considerations section.

In a fusion device environment, the device performing the leaking may not even be the direct next hop from the border. It may be several physical hops away. In this environment, the VRFs must be maintained, commonly using VRF-lite, from the border to the device ultimately performing the route leaking. In this deployment type, the next-hop from the border is VRF-aware along with the devices in the data path towards the fusion.


Non-VRF-Aware Peer

This deployment type begins with VRF-lite configured on the border node, and the peer configured, though not VRF-aware. For each VN that is handed off on the border node, a corresponding interface is configured on the peer device in the global routing table. This deployment option is commonly used when the fabric site hands off to a WAN circuit, ISP, an MPLS CE or PE device, other upstream routing infrastructure.

This deployment type is common in WAN infrastructure. If this next-hop peer is an MPLS CE, routes are often merged into a single table to reduce the number of VRFs to be carried across the backbone, generally reducing overall operational costs. If the next-hop peer is an MPLS PE or ISP equipment, it is outside of the administrative domain of the fabric network operator. The result is that there is little flexibility in controlling the configuration on the upstream infrastructure. Many times, ISPs have their own peering strategies and themselves are presenting a Layer 3 handoff to connected devices.

Non-VRF aware means that peer router is not performing VRF-lite. It may have the functionality to support VRFs, but it is not configured with corresponding fabric VRFs the way a VRF-Aware peer would be. The non- VRF aware peer is commonly used to advertise a default route to the endpoint-space in the fabric site.

The result is the VNs from the fabric site are merged into a single routing table (GRT) on the next-hop peer. Merging routes into a single table is a different process than route leaking. This deployment type does use the colloquial moniker of fusion router.

The challenge with merged tables is the potentiality of East-West communication across the North-South link. Merging the VRFs into a common routing table is best accomplished with a firewall. Firewalls are policy- oriented devices that align well with the segmentation provided through the LISP/VXLAN fabric solution.

It is not always possible to use a firewall in environments that use route-table merging such as with WAN circuits listed above. However, degrees of precaution and security can be maintained, even without a firewall. For example, specific Security Group tags (SGTs) or port-based ACLs can limit and prevent East-West communication. Further protection can be added by sinkhole routing. This is done manually on the border node, for each VRF, by pointing the aggregate prefixes for each other VRF to Null0.

In the simplified topology in Figure 32 below, the border node is connected to a non-VRF-aware peer with each fabric VNs, and their associated subnet are represented by a color. This type of connection effectively merges the fabric VN routing tables onto a single table (generally GRT) on the peer device. By route sinking as described above, the East-West communication between the VNs can be prevented across the North-South link between the border node and its peer.

Figure 32. Simplified Route Sinking Example

Slide33.JPG

 

 


Firewall Peer

If the fabric VNs need to merge to a common routing table, a policy-oriented device such as a firewall should be considered as an upstream peer from the fabric border nodes. Common use cases for a firewall peer include Internet access, access to data center prefixes, WAN connectivity, or Inter-VN communication requirements.

Tech tip

In most deployments, endpoints, users, or devices that need to directly communicate with each other should be placed in the same overlay virtual network. Some networks may have specific requirements for VN-to-VN communication, though these are less common. VN to VN requirements are often seen during mergers of companies or in some corporate or government structures or similar multi-tenant environment where each agency, tenant, or division is required to have their own VN-space.

A firewall can be used to provide stateful inspection for inter-VN communication along with providing Intrusion Prevent System (IPS) capabilities, Secure Endpoint granular Application Visibility and Control (AVC), and even URL filtering. Firewalls such as Cisco ASA and Cisco Secure Firewall also provide a very rich reporting capability with information on traffic source, destination, username, group, and firewall action with guaranteed logging of permits and drops.

Firewalls can be deployed as a cluster (multiple devices acting as a single logical unit), as an HA pair (commonly Active/Standby), or even as a standalone device. While firewalls do not generally have VRF capabilities, they have other method for providing the same general type of segmentation provided by VRFs. These include contexts, interface-specific ACL, and security-levels (ASA), instances, and security zones (FTD).


External Connectivity Design Considerations

The same considerations and conventions apply to external connectivity as they do to connections between layers in Enterprise Campus Architecture: build triangles, not squares, to take advantage of equal-cost redundant paths for the best deterministic convergence. Border nodes should be deployed in pairs and should each connect to a pair of upstream devices. Border nodes should have a crosslink between each other. If the upstream infrastructure is within the administrative domain of the network operator, these devices should be crosslinked to each other. Border nodes of the same type, such as internal and external, should be fully meshed.

In some deployments, the upstream device from border nodes may be a single logical unit represented by two or more devices such as VSS, SVL, or even a firewall cluster. To build triangle topologies, the border nodes should be connected to each device in the logical unit.

Tech tip

Figures 23-26 below show the peer device as a StackWise Virtual device, although the failover scenarios represented are also applicable to Active-Standby Firewalls and other HA upstream pairs.

In Figure 23 below, the physical topology uses squares to connect the devices. Each border node is connected to only one member of upstream logical peer. The border nodes are crosslinked to each other which provides an indirect and non-optimal forwarding path in the event of an upstream link failure. The resulting logical topology is an incomplete triangle.

Figure 23. Incomplete Triangle – Logical Topology

Slide35.JPG

 

 

In Figure 24 below, the physical topology uses triangles to connect the devices. Each border node is connected to each member of the upstream logical peer. The border nodes are crosslinked to each other. The resulting

logical topology is the same as the physical, and a complete triangle is formed. This is the recommended approach.

Figure 24. Complete Triangle – Logical Topology

Slide36.JPG

 

 

 

 

Figure 25 below shows a pair of border node connected to a StackWise Virtual upstream peer. If the link to one StackWise member has a failure scenario, IP reachability still exists, but Border Node #1 must traverse Border Node #2 to reach destinations beyond the upstream peer. And while IP reachability still exists, it is an inefficient forwarding path that requires VRF-awareness (VRF-lite) between the redundant borders to achieve. This topology example represents a single point of failure akin to having a single upstream device from the redundant border nodes.

Figure 25. Square Topology Failover Scenario

Slide37.JPG

 

 

 

In contrast, as shown in Figure 26 below, if the border nodes are connected to both StackWise peers, even in the event of a single member failure, each border node will still have an optimal, redundant forwarding path.

Figure 26. Triangle Topology Failover Scenario

Slide38.JPG

 

 

 

Security Policy Design Considerations

This section is organized into the following subsections:

Section Subsection
Security Policy Design Considerations Aspects and Design Criteria

A one-size-fits-all security design is not desirable—security requirements vary by organizations. Security designs are driven by information security policies and legal compliance. The planning phase for a security design is key to ensuring the right balance of security and user experience. The following aspects should be considered when designing security policy for the LISP/VXLAN fabric network:

  • Openness of the network—Some organizations allow only organization-issued devices in the network, and some support a Bring Your Own Device (BYOD) approach. Alternatively, user choice can be balanced with allowing easier-to-manage endpoint security by deploying a Choose Your Own Device (CYOD) model in which a list of IT-approved endpoints is offered to the users for business use. An identity-based approach is also possible in which the network security policies deployed depend on the device ownership. For example, organization-issued devices may get Security Group access, while personal devices may get Internet-only access.

  • Identity management—In its simplest form, identity management can be a username and password used for authenticating users. Adding embedded security functions and application visibility in the network provides telemetry for advanced policy definitions that can include additional context such as physical location, device used, type of access network (wired, wireless, VPN), application used, and time of day.

  • Authentication, Authorization, and Accounting (AAA) policies—Authentication is the process of establishing and confirming the identity of a client requesting access to the network. Authorization is the process of authorizing access to some set of network resources. Accounting is process of recording what was done and accessed by the client.For unified experience for wired and wireless endpoints, AAA policies in LISP/VXLAN fabric are enforced at the access layer (edge nodes) with the use of SGACLs for segmentation within VNs and dynamic VLAN assignment for mapping endpoints into VNs. Event logs, ACL hit counters, RADIUS accounting, and similar standard accounting tools are available to enhance visibility.

  • Endpoint security—Endpoints can be infected with malware, compromising data, and creating network disruptions. Malware detection, endpoint management, and data exports from the network devices provide insight into endpoint behavior. Tight integration with security appliances such as Cisco Adaptive Security Appliances (ASA) and Cisco Secure Firewall and analytics platforms such as Cognitive Intelligence enables the network to have the intelligence to quarantine and help remediate compromised devices.

  • Data integrity and confidentiality—Network segmentation using VNs can control access to applications such as separating employee transactions from IoT traffic.

  • Network device security—Hardening security of network devices is essential. The use of the secure device management options, such as enabling device authentication using TACACS+ and disabling unnecessary services, are best practices to ensure the network devices are secured.

Enabling Security Group segmentation within each virtual network allows for simplified hierarchical network policies. Network-level policy scopes of isolated control and data planes are possible using VNs, while group- level policy scopes are possible using SGTs within VNs, enabling common policy application across the wired and wireless fabric.

SGTs tag endpoint traffic based on a role or function within the network such that the traffic is subject to role- based policies or SGACLs centrally defined within ISE which references Active Directory, for example, as the identity store for user accounts, credentials, and group membership information. Endpoints can be classified based on that identity store information and can be assigned to an appropriate Security Group. These Security Groups can then be used to create segmentation policies and virtual network assignment rules.

SGT information is carried across the network in several forms:

  • Inside the LISP/VXLAN fabric —The LISP/VXLAN fabric header transports SGT information. Fabric edge nodes and border nodes can enforce SGACLs to enforce the security policy.

  • Outside the fabric on a device with Cisco Security Group capability—Inline devices with Cisco Security Group capability carry the SGT information in a CMD header on the Layer 2 frame. This is the recommended mode of transport outside the LISP/VXLAN fabric network.

  • Outside the fabric over devices without Cisco Security Group capability—SXP allows the control plane communication of SGT to IP mappings over a TCP connection. This can be used to communicate SGTs over network devices that do not support SGT inline tagging. We can also use pxGrid to propagate the SGTs from Cisco Secure Firewall to the devices.

Multidimensional Considerations

This section is organized into the following subsections:

Section Subsection
Multidimensional Considerations

Number of Users

Shared Services

VRF-Aware Peers (Fusion Devices)

WAN and Internet Connectivity

End-to-End Macro Segmentation

End-to-End Micro Segmentation

Site Survivability

High Availability

An LISP/VXLAN fabric network begins with a foundation of the Cisco Enterprise Architecture Model with well-designed and planned hierarchical network structures that include modular and extensible network blocks as discussed in the LAN Design Principles section. On this foundation, the network is designing and configured using the Layer 3 routed access model.

While individual sites can have some design and configuration that is independent from other locations, this design and configuration must consider how the site becomes part of the larger campus network including other fabric sites, non-fabric sites, shared services, data center, WAN, and Internet. No element, consideration, or fabric site should be viewed in isolation, and an end-to-end view of the network must be considered.

Beyond the business needs, business drivers, and previous listed Design Considerations, additional technical factors must be considered. The results of these technical considerations craft the framework for the topology and equipment used in the network. These factors are multi-dimensional and must be considered holistically. The design strategy for LISP/VXLAN fabric is to maximize site size while minimizing site count. Each of the factors below could drive the need to deploy multiple, smaller fabric sites rather than one larger one.

Number of Users

The most significant factor in the selection of equipment and topology for a site, apart from existing wiring, is total number of wired and wireless clients in that location. This will determine the number of physical switch ports and access points required which will determine the need for three-tier or two-tier network designs. The number of clients may be small enough that the network is composed of a switch stack or large enough to cover multiple buildings with many thousands of endpoints.

Shared Services

Services such as DHCP, DNS, ISE, and WLCs are required elements for clients in a LISP/VXLAN fabric network. Services are commonly deployed in one of three ways.

  • Fabric Site Local—For survivability purposes, a services block may be established at each fabric site location. Local services ensure that these critical services are not sent across the WAN/MAN/Internet and ensure the endpoints are able to access them, even in the event of congestion or unavailability of the external circuit. However, this may drive the need for VRF-aware peering devices to fuse routes from the fabric overlay to shared services.

  • Centralized within the Deployment—In locations distributed across a WAN, services are often deployed at on-premises data centers. These data centers are commonly connected to the core or distribution layers of a centralized location such as a headquarters. Traffic is sent from the remote and branch sites back to the central location, and then directed towards the necessary services.

  • Both Centralized and Fabric-Site Local—This is a hybrid of the two approaches above. For most fabric sites, services are centralized. Specific fabric sites with a need for services connectivity independent of the status of the WAN circuit use local services.

VRF-Aware Peer (Fusion Devices)

While not a specific reason factor in the decision to deploy multiple fabric sites, shared services must be considered as part of the deployment. A VRF-Aware peer (fusion device) is the most common deployment method to provide access to shared services. For fabric sites needing resiliency, high availability, and site survivability independent of WAN status, local shared services are needed. These locations should plan for the use of a services block and VRF-aware peer to provide the fabric endpoint access to these services.

WAN and Internet Connectivity

External Internet and WAN connectivity for a fabric site has a significant number of possible variations. The key design consideration is to ensure the infrastructure has the physical connectivity, routing information, scale, performance, and throughput necessary to connect the fabric sites to the external world.

End-to-End Macro Segmentation (VRFs)

With IP-based transits, due to de-encapsulation of the fabric packet, VN policy

information can be lost. Several approaches exist to carry VN (VRF) information between fabric sites using an IP-based transit. The most straightforward approach is to configure VRF-lite hop-by-hop between each fabric site. While this is the simplest method, it also has the highest degree of administrative overhead. This method is not commonly utilized, as the IP-based infrastructure between fabric sites is generally under the administrative control of a service provider.

If VRF-lite cannot be used end to end, an alternative approach is to carry VRFs is by using other means such as DMVPN, GRE, IPsec. These designs begin with a VRF-lite handoff on the border node. The VRF is associated with an 802.1Q VLAN to maintain the segmentation construct. This design supports switching platform as the border node. However, the peer device needs to be a routing platform to support the applicable protocols.

  • Border Node with GRE Peer—A VRF is handed off via a VLAN to a GRE tunnel endpoint router. On the tunnel endpoint router, one GRE tunnel is configured per fabric VN.

  • Border Node with DMVPN Peer—A VRF is handed off via a VLAN to a DMVPN router. On the DMVPN router, one DMVPN cloud is configured per fabric VN.

  • Border Node with IPSec Peer—A VRF is handed off via a VLAN to an IPSec router. On the IPSec router, one IPsec tunnel is configured per fabric VN.

  • Border Node with MP-BGP Peer— A VRF is handed off via a VLAN to a peer supporting multiprotocol BGP such as MPLS provider. BGP needs a VRF-Aware data plane such as MPLS to have a mechanism to carry the VRF attributes.

  • Border Node with Cisco SD-WAN Peer — A VRF is handed off to a IOS XE SD-WAN device. BGP VRF-lite handoff is between Border node and SD-WAN device.

End-to-End Micro segmentation (SGTs)

With IP-based transit, due to the de-encapsulation of the fabric packet, SGT policy information can be lost. Two approaches exist to carry SGT information between fabric sites using an IP-based transit, inline tagging and SXP.

Inline Tagging

Inline tagging is the process where the SGT is carried within a special field known as CMD (Cisco Meta Data) that can be inserted in the header of the Ethernet frame. This changes the EtherType of the frame to 0x8909. If the next-hop device does not understand this EtherType, the frame is assumed to be malformed and is discarded. Inline tagging can propagate SGTs end to end in two different ways.

  • Hop by Hop—Each device in the end-to-end chain would need to > support inline tagging and propagate the SGT.

  • Preserved in Tunnels—SGTs can be preserved in CMD inside of GRE > encapsulation or in CMD inside or IPsec encapsulation.

SGT Exchange Protocol over TCP (SXP)

A second design option is to use SXP to carry the IP-to-SGT bindings between sites. SXP is used to carry SGTs across network devices that do not have support for Inline Tagging or if the tunnel used is not capable of carrying the tag.

SXP has both scaling and enforcement location implications that must be considered. Between fabric sites, SXP can be used to enforce the SGTs at either the border nodes or at the routing infrastructure north bound of the border. If enforcement is done at the routing infrastructure, CMD is used to carry the SGT information inline from the border node.

If enforcement is done on the border node, a per-VRF SXP peering must be made with each border node to ISE. A common way to scale SXP more efficiently is to use SXP domains. A second alternative is to peer the border node with a non-VRF-Aware Peer and merge the routing tables. ISE then makes a single SXP connection to each of these peers. Another alternative approach is to use SXPv5 where we can carry multiple VRF over a single SXP connection.

Tech tip

For additional details on deployment scenarios, SGTs over GRE and VPN circuits, and scale information, please see the SD- Access Segmentation Design Guide.

Tech tip

For additional details on SXPv5, please see the Security Group Policy SXPv5 Guide

Site Survivability

In the event that the WAN and MAN connections are unavailable, any service accessed across these circuits are unavailable to the endpoints in the fabric. The need for site survivability is determined by balancing the associated costs of the additional equipment and the business drivers behind the deployment while also factoring in the number of impacted users at a given site. Designing a LISP/VXLAN fabric network for complete site survivability involves ensuring that shared services are local to every single fabric site. Generally, a balance between centralized and site-local services is used. If a given fabric site has business requirements to always be available, it should have site-local services. Other fabric sites without the requirement can utilize centralized services for the fabric.

 

High Availability

High availability compliments site survivability. A site with single fabric border, control plane node, or wireless controller risks single failure points in the event of a device outage. When designing for high availability in a LISP/VXLAN network, it is important to understand that redundant devices do not increase the overall scale.

Redundant control plane nodes and redundant border nodes operate in an active-active method, and Fabric WLCs operate as active-standby pairs.

LISP/VXLAN Site Reference Models

This chapter is organized into the following sections:

Chapter Section
LISP/VXLAN Site Reference Models

Design Strategy

Fabric in a Box Site Reference Model

Very Small Site Reference Model

Small Site Reference Model

Medium Site Reference Mode

Large Site Reference Model

Designing Cisco LISP/VXLAN fabric site has flexibility to fit many environments, which means it is not a one- design-fits-all proposition. The scale of a fabric can be as small a single switch or switch stack or as big as one or more three-tier campus deployments. LISP/VXLAN fabric topologies should follow the same design principles and best practices associated with a hierarchical design, such splitting the network into modular blocks and distribution of functions.

Design elements should be created that can be replicated throughout the network by using modular designs. In general, LISP/VXLAN fabric topologies should be deployed as spoke networks with the fabric border node as the exit point hub for the spokes which are the access switches operating as edge nodes. As networks grow, varied physical topologies are used to accommodate requirements for specialized network services deployment.

Fabric Site Sizes – Design Strategy

A practical goal for LISP/VXLAN fabric designs is to create larger fabric sites rather than multiple, smaller fabric sites. The design strategy is to maximize fabric site size while minimizing total site count. Some business requirements will necessitate splitting locations into multiple sites such as creating a fabric site for an Emergency Room (ER) that is separate from the fabric site that is represented by the remainder of the hospital. The multidimensional factors of survivability, high availability, number of endpoints, services, and geography are all factors that may drive the need for multiple, smaller fabric sites instead of a single large site. To help aid in design of fabric sites of varying sizes, the Reference Models below were created.

Fabric Site Reference Models

In deployments with physical locations, customers use different templates for each of the different site types such as a large branch, a regional hub, headquarters, or small, remote office. The underlying design challenge is to look at existing network, deployment, and wiring, and propose a method to layer LISP/VXLAN fabric sites in these areas. This process can be simplified and streamlined by templatizing designs into reference models.

The templates drive understanding of common site designs by offering reference categories based on the multidimensional design elements along with endpoint count to provide guidelines for similar site size designs. The numbers are used as guidelines only and do not necessarily match maximum specific scale and performance limits for devices within a reference design.

  • Fabric in a Box site—Uses Fabric in a Box to cover a single fabric site, with resilience supported by switch stacking or StackWise Virtual; designed for less than 200 endpoints, less than 5 VNs, and less than 40 APs; the border, control plane, edge, and wireless functions are co-located on a single redundant platform.

  • Very small site— Covers a single office or building; designed to support less than 2,000 endpoints, less than 8 VNs, and less than 100 APs; the border is co-located with the control plane function on one or two devices and uses embedded wireless or optionally hardware WLCs.

  • Small site—Covers a single office or building; designed to support less than 10,000 endpoints, less than 32 VNs, and less than 200 APs; the border is co-located with the control plane function on one or two devices and a separate wireless controller has an optional HA configuration.

  • Medium site—Covers a building with multiple wiring closets or multiple buildings; designed to support less than 25,000 endpoints, less than 50 VNs, and less than 1,000 APs; the border is distributed from the control plane function using redundant devices, and a separate wireless controller has an HA configuration.

  • Large site—Covers a large building with multiple wiring closets or multiple buildings; designed to support less than 50,000 endpoints, less than 64 VNs, and less than 2,000 APs; multiple border exits are distributed from the control plane function on redundant devices, and a separate wireless controller has an HA configuration.

Each fabric site includes a supporting set of control plane nodes, edge nodes, border nodes, and wireless LAN controllers, sized appropriately from the listed categories. ISE Policy Service Nodes are also distributed across the sites to meet survivability requirements.

Fabric in a Box Site Reference Model

The Fabric in a Box Site Reference Model should target less than 200 endpoints. The central component of this design is a switch stack or StackWise Virtual operating in all three fabric roles: control plane node, border node, and edge node. For switch stack Fabric in a Box deployments, LISP/VXLAN fabric Embedded Wireless is used to provide site local WLC functionality. The site may contain an ISE PSN depending on the WAN/Internet circuit and latency.

Use the table below to understand the guidelines to stay within for similar site design sizes. The numbers are used as guidelines only and do not necessarily match specific limits for devices used in a design of this site size.

Table 1. Fabric in a Box Site Guidelines (Limits may be different)

 

Endpoints, target fewer than 200
Control plane nodes, co-located 1
Border nodes, co-located 1
Virtual networks, target fewer than 5
IP pools, target fewer than 8
Access points, target fewer than 40

Figure 27. Physical Topology – Fabric in a Box Site Design

Slide39.JPG

 

 

Fabric in a Box Site Considerations

Due to the smaller number of endpoints, and so implied lower impact, high availability and site survivability are not common requirements for a Fabric in a Box design. As with all the reference designs, site-local services of

DHCP, DNS, WLCs, and ISE can provide resiliency and survivability although at the expense of increased complexity and equipment such as a services block.

If shared services are deployed locally, the peer device is commonly a switch directly connected to the Fabric in a Box with services deployed as virtual machines on Cisco UCS C-Series Server. An alternative is to deploy a UCS E-series blade servers on the routing infrastructure to virtualize the shared services.

High availability in this design is provided through StackWise-480 or StackWise Virtual which both combine multiple physical switches into a single logical switch. StackPower is used to provide power redundancy between members in a switch stack. StackWise Virtual deployments have power redundancy by using dual power supplies in each switch. If a chassis-based switch is used, high availability is provided through redundant supervisors and redundant power supplies. To support power redundancy, available power supplies would need to be redundant beyond the needs of the switch to support power chassis, supervisor, and line cards.

Wireless LAN controllers can be deployed as physical units directly connected to the Fabric in a Box or deployed as the embedded Catalyst 9800 controller. When using the embedded Catalyst 9800 with a switch stack or redundant supervisor, AP, and Client SSO (Stateful Switch Over) are provided automatically.

When using stacks, links to the upstream routing infrastructure should be from different stack members. Ideally, the uplinks should be from the member switches rather than the stack master. With chassis switches, links should be connected through different supervisors. To prepare for border node handoff configuration along with having initial IP reachability, SVIs and trunk links are commonly deployed between the small site switches and the upstream routing infrastructure.

The Catalyst 9300 Series in a stack configuration with the embedded Catalyst 9800 Series wireless LAN controller capabilities is an optimal platform in this design. Other available platforms such as the Catalyst 9500 Series can be deployed as StackWise Virtual and can provide connectivity options such as SFP+ (10 Gigabit Ethernet) and multi-chassis redundancy capabilities.

Very Small Site Reference Model

The Very Small Site Reference Model should target less than 2,000 endpoints. The physical network is usually a two-tier collapsed core/distribution with an access layer servicing several wiring closets. Rather than co-locating all roles in one device, the Very Small Site Reference Model provides added resiliency and redundancy along with a larger number of endpoints by separating the edge node role onto dedicated devices in the access layer. The border and control plane node are co-located in the collapsed core layer. For LISP/VXLAN fabric Wireless, the embedded WLC is provisioned on one of the co-located border and control plane nodes. Optionally, a virtual or hardware based WLC is used.

Use the table below to understand the guidelines to stay within for similar site design sizes. The numbers are used as guidelines only and do not necessarily match specific limits for devices used in a design of this site size. The target maximum number of endpoints is based on approximately ~50% of the number endpoints supported by the Catalyst 9800 Embedded Wireless controller as documented on the Cisco Access Point and Wireless Controller Selector.

Table 2. Very Small Site Reference Guidelines (Limits may be different)

 

Endpoints, target fewer than 2,000
Fabric nodes, target fewer than 50
Control plane nodes, co-located 2
Border nodes, co-located 2
Virtual networks, target fewer than 8
IP pools, target fewer than 20
Access points, target fewer than 100

Figure 28. Physical Topology – Very Small Site Design

Slide40.JPG

 

 

Very Small Site Considerations

For very small deployments, a LISP/VXLAN fabric site is implemented using a two-tier design. The same design principles for a three-tier network are applicable, though there is no need for a distribution layer (intermediate nodes). In a very small site, high availability is provided in the fabric nodes by co-locating the border node and control plane node functionality on the collapsed core switches and deploying these as a pair. For both resiliency and alternative forwarding paths in the overlay and underlay, the collapsed core switches should be directly connected to each other with a crosslink.

Provided there are less than 200 APs and 4,000 clients, LISP/VXLAN fabric Embedded wireless can be deployed along with the co-located border node and control plane node functions on a collapsed core switch. For high- availability for wireless, a hardware or virtual WLC should be used.

Small Site Design Reference Model

The Small Site Reference Model covers a building with multiple wiring closets or multiple buildings and typically has less than 10,000 endpoints. The physical network is usually a two-tier collapsed core/distribution with an access layer.

Use the table below to understand the guidelines to stay within for similar site design sizes. The numbers are used as guidelines only and do not necessarily match specific limits for devices used in a design of this site size.

Table 3. Small Site Guidelines (Limits may be different)

 

Endpoints, target fewer than 10,000
Fabric nodes, target fewer than 75
Control plane nodes 2
Border nodes 2
Virtual networks, target fewer than 32
IP pools, target fewer than 100
Access points, target fewer than 200

Figure 29. Physical Topology – Small Site Reference Design

Slide41.JPG

 

 

Small Site Considerations

For smaller deployments, a LISP/VXLAN fabric site is implemented using a two-tier design. The same design principles for a three-tier network applicable, though there is no need for an aggregation layer (intermediate nodes). In a small site, high availability is provided in the fabric nodes by co-locating the border node and control plane node functionality on the collapsed core switches and deploying these as a pair. For both resiliency and alternative forwarding paths in the overlay and underlay, the collapsed core switches should be directly connected to each other with a crosslink.

The client and access point count calls for use of dedicated WLCs either in hardware or virtual machines. To enable highly available links for WLC through physical connectivity, a services block is deployed. The WLCs are connected to the services block switch through Layer 2 port-channels to provide redundant interfaces. The services block is a switch stack or SVL that is connected to both collapsed core switches through Layer 3 routed

links. This services block is deployed as a VRF-aware peer if DHCP/DNS and other shared services are site- local.

Medium Site Design Reference Model

The Medium Site Reference Model covers a building with multiple wiring closets or multiple buildings and is designed to support less than 25,000 endpoints. The physical network is usually a three-tier network with core, distribution, and access layers. It may even contain a routed super-core that aggregates multiple buildings and serves as the network egress point to the WAN and Internet. The border and control plane node functionality are provisioned on separate devices rather than co-locating.

Use the table below to understand the guidelines to stay within for similar site design sizes. The numbers are used as guidelines only and do not necessarily match specific limits for devices used in a design of this site size.

Table 4. Medium Site Guidelines (Limits may be different)

 

Endpoints, target fewer than 25,000
Fabric nodes, target fewer than 450

Control plane nodes

(Limit to 2 Enterprise + 2 Guest for LISP/VXLAN fabric Wireless)

 2-4
Border nodes 2
Virtual networks, target fewer than 50
IP pools, target fewer than 200
Access points, target fewer than 1,000

Figure 30. Physical Topology – Medium Site Reference Design

Slide42.JPG

 

 

Medium Site Considerations

In a medium site, high availability is provided in the fabric nodes by dedicating devices as border nodes and control plane nodes rather than collocating the functions together. For both resiliency and alternative forwarding paths in the overlay and underlay, all devices within a given layer, with the exception of the access layer, should be crosslinked to each other. Multiple distribution blocks do not need to be cross- connected to each block, though should cross-connect to all distribution switches within a block. Dedicated control plane nodes are generally connected to the core switches so that they are highly available for any edge node within the various distribution blocks. For optimal forwarding and redundancy, they should have connectivity through both cores, and if interfaces and fiber is available, crosslink to each other though this is not a requirement.

Physical WLC should be deployed to support the wireless user scale. To enable high availability a WLC HA- SSO pair is deployed with redundant physical connectivity to a services block using Layer 2 port-channels. The WLCs should be connected to each other through their Redundancy Port in accordance with the Tech tip from the Services Block section above. The services block is commonly implemented with fixed configuration switches operating in VSS or StackWise Virtual and connected to the core through Layer 3 routed links. This services block is deployed as a VRF-aware peer if DHCP/DNS and other shared services are site-local.

Large Site Design Reference Model

The Large Site Reference Model covers a building with multiple wiring closets or multiple buildings. The physical network is a three-tier network with core, distribution, and access and is designed to support less than 40,000 endpoints. This network is large enough to require dedicated services exit points such as a dedicated data center, shared services block, and Internet services.

Use the table below to understand the guidelines to stay within for similar site design sizes. The numbers are used as guidelines only and do not necessarily match specific limits for devices used in a design of this site size.

Table 5. Large Site Guidelines (Limits may be different)

 

Endpoints, target fewer than 50,000
Fabric nodes, target fewer than 750

Control plane nodes

(Limit to 2 Enterprise + 2 Guest for LISP/VXLAN fabric Wireless)

 2-4

Border nodes

(2 as Internal and 2 as External)

 2-4
Virtual networks, target fewer than 64
IP pools, target fewer than 450
Access points, target fewer than 2,000

Figure 31. Physical Topology – Large Site Reference Design

Slide44.JPG

 

 

 

 

Large Site Considerations

The large site design is commonly the headquarters (HQ) location in a multiple-fabric site deployment. The enterprise edge firewall (perimeter firewall) is usually deployed at this location, and Internet traffic from remote sites is tunnel back to this site to be processed by the perimeter security stack before being forwarded to the Internet.

Control plane nodes and border nodes should be dedicated devices deployed as redundant pairs. Dedicated control plane nodes should be connected to each core switch to provide resiliency and to have redundant forwarding paths. If interfaces and fiber are available, crosslink the control plane nodes to each other though this is not a requirement; it simply provides another underlay forwarding path.

A wireless LAN controller HA-SSO pair is deployed with redundant physical connectivity to a services block using Layer 2 port-channels. The WLCs should be connected to each other through their Redundancy Ports in accordance with the Tech tip from the Services Block section above. The services block is commonly part of the on-premises data center network. At this headquarters location, the data center core is connected to either the campus core or the distribution switches to provide reachability to services and applications.

Dedicated internal border nodes are commonly used to connect the fabric site to the data center core while dedicated external border nodes are used to connect the site to the MAN, WAN, and Internet. The Large Site may contain the DMZ where the dedicated Anchor fabric border and control plane nodes for Guest Wireless are deployed.

 

Appendices

This chapter is organized into the following sections:

Chapter Section
Appendix A LISP/VXLAN Fabric Protocols Deep Dive
Appendix B References Used in this Guide
Appendix C Glossary of Terms and Acronyms

Appendix A – LISP/VXLAN Fabric Protocols

This section is organized into the following subsections:

Section Subsection
LISP/VXLAN Fabric Protocols

Fabric Data Plane

Fabric Control Plane

Fabric Data Plane

RFC 7348 defines the use of virtual extensible LAN (VXLAN) as a way to overlay a Layer 2 network on top of a Layer 3 network. Each overlay network is called a VXLAN segment and is identified using a 24-bit VXLAN network identifier, which supports up to 16 million VXLAN segments.

The LISP/VXLAN fabric uses the VXLAN data plane to provide transport of the full original Layer 2 frame and additionally uses LISP as the control plane to resolve endpoint-to-location (EID-to-RLOC) mappings. The LISP/VXLAN fabric replaces sixteen (16) of the reserved bits in the VXLAN header to transport up to 64,000 SGTs using a modified VXLAN-GPO (sometimes called VXLAN-GBP) format described in https://tools.ietf.org/html/draft-smith-vxlan-group-policy-04.

The Layer 3 VNI maps to a virtual routing and forwarding (VRF) instance for Layer 3 overlays, whereas a Layer 2 VNI maps to a VLAN broadcast domain, both providing the mechanism to isolate data and control plane to each individual virtual network. The SGT carries group membership information of users and provides data-plane segmentation inside the virtualized network.

Figure 33 VXLAN-GBP header

Picture34.jpg

 

Fabric Control Plane

RFC 6830 through RFC 6836 along with later RFCs define LISP as a network architecture and set of protocols that implement a new semantic for IP addressing and forwarding. In traditional IP networks, the IP address is used to identify both an endpoint and its physical location as part of a subnet assignment on a device. In a LISP- enabled network, an IP address or MAC address is used as the endpoint identifier for an endpoint, and an additional IP address is used as an RLOC to represent the physical network device the endpoint is connected directly to or directly through such as with an access point. The Loopback 0 address of the network device is used as the RLOC address. The EID and RLOC combination provide the necessary information for traffic forwarding. The RLOC address is part of the underlay routing domain, and the EID can be assigned independently of the location.

The LISP architecture requires a mapping system that stores and resolves EIDs to RLOCs. This is analogous to using DNS to resolve IP addresses for host names. EID prefixes (either IPv4 addresses with /32 mask, MAC Address, or IPv6 Addresses with /128 masks) are registered with the map server along with their associated RLOCs. When sending traffic to an EID, a source RLOC queries the mapping system to identify the destination RLOC for traffic encapsulation. As with DNS, a local node probably does not have the information about everything in a network but instead asks for the information only when local hosts need it to communicate (pull model). This information is then cached for efficiency.

It is helpful to understand how these technologies support the deployment goals. Included benefits provided by the LISP architecture are:

  • Network virtualization—A LISP Instance ID (IID) is used to maintain independent VRF and VLAN topologies. From a data-plane perspective, the LISP Instance ID maps to either a Layer 2 or Layer 3 VNI.

  • Subnet stretching—A single subnet can be extended to exist at multiple RLOCs. The separation of EID from RLOC enables the capability to extend subnets across different RLOCs. As a result of the availability of the Anycast Gateway across multiple RLOCs, the client configuration (IP address, subnet, and gateway) can remain unchanged, even as the client moves across the stretched subnet to different physical attachment points.

  • Smaller routing tables—Only RLOCs need to be reachable in the global routing table for communication within a fabric site. Local EIDs (connected endpoints) are cached at the local node while remote EIDs (endpoints connected to or through other fabric devices) are learned through conversational learning. Conversational learning is the process of populating forwarding tables with only endpoints that are communicating through the node. This allows for efficient use of forwarding tables.

Appendix B – References Used in Guide

LISP VXLAN Fabric Configuration Guide, Cisco IOS XE Cupertino 17.9.x (Catalyst 9300 Switches)

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/17-9/configuration_guide/lisp_vxlan/b-179-lisp-vxlan-fabric-cg.html

LISP/VXLAN Fabric Ansible Playbook

https://developer.cisco.com/codeexchange/github/repo/sandeep3in/lisp_vxlan_ansible

Automating SDA Fabric Creation with Ansible

https://community.cisco.com/t5/networking-knowledge-base/automating-sda-fabric-creation-with-ansible/ta-p/4909709

Using Ansible LISP VXLAN Fabric YouTube Video

https://www.youtube.com/watch?v=fsxlo9DTJUo

LISP VXLAN Fabric Troubleshooting Guide

https://www.cisco.com/c/en/us/support/docs/troubleshooting/220361-troubleshoot-lisp-vxlan-fabric-issues.html

Cisco DNA Center & ISE Management Infrastructure Deployment Guide

https://www.cisco.com/c/en/us/td/docs/solutions/CVD/Campus/cisco-dnac-ise-deploy-guide.html

SD-Access Fabric Provisioning Prescriptive Deployment Guide

https://www.cisco.com/c/en/us/td/docs/solutions/CVD/Campus/cisco-sda-design-guide.html

Cisco SD-Access Fabric Resources

https://community.cisco.com/t5/networking-knowledge-base/cisco-sd-access-fabric-resources/ta-p/4196271

ISE Performance & Scale

https://www.cisco.com/c/en/us/td/docs/security/ise/performance_and_scalability/b_ise_perf_and_scale.html

Hierarchical Network Design Overview - Cisco Networking Academy:

https://www.ciscopress.com/articles/article.asp?p=2202410&seqNum=4

Enterprise Campus 3.0 Architecture: Overview and Framework

https://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Campus/campover.html

Campus Network for High Availability Design Guide

https://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Campus/HA_campus_DG/hacampusdg.html

High Availability Campus Network Design–Routed Access Layer using EIGRP or OSPF

https://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Campus/routed-ex.html

Campus Network for High Availability Design Guide

https://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Campus/HA_campus_DG/hacampusdg.html#wp1107578

High Availability Campus Network Design - Routed Access Layer using EIGRP or OSPF System Assurance Guide

https://www.cisco.com/c/dam/en/us/td/docs/nsite/campus/ha_campus_routed_access_cvd_ag.pdf

High Availability SSO Deployment Guide for Cisco Catalyst 9800 Series Wireless Controllers, Cisco IOS XE Bengaluru 17.6

https://www.cisco.com/c/dam/en/us/td/docs/wireless/controller/9800/17-6/deployment-guide/c9800-ha-sso-deployment-guide-rel-17-6.pdf

SD-Access Segmentation Design Guide

https://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Campus/CVD-Software-Defined-Access-Segmentation-Design-Guide-2018MAY.pdf

Group-Based Policy SXPv5 Guide

https://community.cisco.com/t5/security-knowledge-base/group-based-policy-sxpv5-guide/ta-p/4705541

Appendix C – Acronym Glossary

AAA—Authentication, Authorization, and Accounting

ACL—Access-Control List

AD—Microsoft Active Directory

AFI—Address Family Identifier

AMP—Cisco Advanced Malware Protection

AP—Access Point

API—Application Programming Interface

ASA—Cisco Adaptative Security Appliance ASM—Any-Source Multicast (PIM)

Auto-RP—Cisco Automatic Rendezvous Point protocol (multicast)

AVC—Application Visibility and Control BFD—Bidirectional Forwarding Detection BGP—Border Gateway Protocol

BMS—Building Management System

BSR—Bootstrap Router (multicast)

BYOD—Bring Your Own Device

CAPWAP—Control and Provisioning of Wireless Access Points Protocol

CDP—Cisco Discovery Protocol CEF—Cisco Express Forwarding CMD—Cisco Meta Data

CPU—Central Processing Unit

CUWN—Cisco Unified Wireless Network CVD—Cisco Validated Design

CYOD—Choose Your Own Device

DC—Data Center

DHCP—Dynamic Host Configuration Protocol

DM—Dense-Mode (multicast)

DMVPN—Dynamic Multipoint Virtual Private Network DMZ—Demilitarized Zone (firewall/networking construct) Cisco Catalyst Center (Formerly DNA—Cisco Digital Network Architecture)

DNS—Domain Name System

DORA—Discover, Offer, Request, ACK (DHCP Process)

ECMP—Equal Cost Multi Path

EID—Endpoint Identifier

EIGRP—Enhanced Interior Gateway Routing Protocol

EMI—Electromagnetic Interference

ETR—Egress Tunnel Router (LISP)

FHR—First-Hop Router (multicast)

FHRP—First-Hop Redundancy Protocol

GbE—Gigabit Ethernet

Gbit/s—Gigabits Per Second (interface/port speed reference)

GRE—Generic Routing Encapsulation GRT—Global Routing Table

HA—High-Availability

HQ—Headquarters

HSRP—Cisco Hot-Standby Routing Protocol

HTDB—Host-tracking Database (LISP/VXLAN control plane node construct) IBNS—Identity-Based Networking Services (IBNS 2.0 is the current version) ICMP— Internet Control Message Protocol

IDF—Intermediate Distribution Frame; essentially a wiring closet.

IEEE—Institute of Electrical and Electronics Engineers

IETF—Internet Engineering Task Force IGP—Interior Gateway Protocol

IID—Instance-ID (LISP)

IOE—Internet of Everything

IoT—Internet of Things

IP—Internet Protocol

IPAM—IP Address Management

IPS—Intrusion Prevention System

IPSec—Internet Protocol Security

ISE—Cisco Identity Services Engine

IS-IS—Intermediate System to Intermediate System routing protocol.

ITR—Ingress Tunnel Router (LISP)

LACP—Link Aggregation Control Protocol LAG—Link Aggregation Group

LAN—Local Area Network

L2 VNI—Layer 2 Virtual Network Identifier; as used in LISP/VXLAN Fabric, a VLAN.

L3 VNI— Layer 3 Virtual Network Identifier; as used in LISP/VXLAN Fabric, a VRF.

LHR—Last-Hop Router (multicast)

LISP—Location Identifier Separation Protocol

MAC—Media Access Control Address (OSI Layer 2 Address)

MAN—Metro Area Network

MEC—Multi-chassis EtherChannel, sometimes referenced as MCEC

MDF—Main Distribution Frame; essentially the central wiring point of the network.

MnT—Monitoring and Troubleshooting Node (Cisco ISE persona)

MOH—Music on Hold

MPLS—Multiprotocol Label Switching MR—Map resolver (LISP)

MS—Map server (LISP)

MSDP—Multicast Source Discovery Protocol (multicast)

MTU—Maximum Transmission Unit

NAC—Network Access Control

NAD—Network Access Device

NAT—Network Address Translation

NSF—Non-Stop Forwarding

OSI—Open Systems Interconnection model OSPF—Open Shortest Path First routing protocol OT—Operational Technology

PAgP—Port Aggregation Protocol

PAN—Primary Administration Node (Cisco ISE persona) PD—Powered Devices (PoE)

PETR—Proxy-Egress Tunnel Router (LISP) PIM—Protocol-Independent Multicast PITR—Proxy-Ingress Tunnel Router (LISP) PnP—Plug-n-Play

PoE—Power over Ethernet (Generic term, may also refer to IEEE 802.3af, 15.4W at PSE)

PoE+—Power over Ethernet Plus (IEEE 802.3at, 30W at PSE)

PSE—Power Sourcing Equipment (PoE)

PSN—Policy Service Node (Cisco ISE persona)

pxGrid—Platform Exchange Grid (Cisco ISE persona and publisher/subscriber service) PxTR—Proxy-Tunnel Router (LISP – device operating as both a PETR and PITR)

QoS—Quality of Service

RADIUS—Remote Authentication Dial-In User Service

RFC—Request for Comments Document (IETF) RIB—Routing Information Base

RLOC—Routing Locator (LISP) RP—Rendezvous Point (multicast) RP—Redundancy Port (WLC)

RPF—Reverse Path Forwarding

RTT—Round-Trip Time

SA—Source Active (multicast)

SAFI—Subsequent Address Family Identifiers (BGP)

SD—Software-Defined

SDA—Cisco Software Defined-Access

SDN—Software-Defined Networking

SFP—Small Form-Factor Pluggable (1 GbE transceiver) SFP+— Small Form-Factor Pluggable (10 GbE transceiver) SGACL—Security-Group ACL

SGT—Scalable Group Tag, sometimes reference as Security Group Tag

SM—Spare-mode (multicast)

SNMP—Simple Network Management Protocol SSID—Service Set Identifier (wireless)

SSM—Source-Specific Multicast (PIM)

SSO—Stateful Switchover

STP—Spanning-tree protocol

SVI—Switched Virtual Interface SVL—Cisco StackWise Virtual SWIM—Software Image Management

SXP—Security Group Tag Exchange Protocol

Syslog—System Logging Protocol

TACACS+—Terminal Access Controller Access-Control System Plus

TCP—Transmission Control Protocol (OSI Layer 4) UCS— Cisco Unified Computing System

UDP—User Datagram Protocol (OSI Layer 4)

UPoE—Cisco Universal Power Over Ethernet (60W at PSE) UPoE+— Cisco Universal Power Over Ethernet Plus (90W at PSE) URL—Uniform Resource Locator

VLAN—Virtual Local Area Network

VN—Virtual Network, analogous to a VRF in LISP/VXLAN

VNI—Virtual Network Identifier (VXLAN)

vPC—virtual PortChannel (Cisco Nexus) VPLS—Virtual Private LAN Service VPN—Virtual Private Network

VPNv4—BGP address family that consists of a Route-Distinguisher (RD) prepended to an IPv4 prefix.

VPWS—Virtual Private Wire Service

VRF—Virtual Routing and Forwarding VSL—Virtual Switch Link (Cisco VSS component) VSS—Cisco Virtual Switching System VXLAN—Virtual Extensible LAN

WAN—Wide-Area Network

WLAN—Wireless Local Area Network (generally synonymous with IEEE 802.11-based networks)

WoL—Wake-on-LAN

xTR—Tunnel Router (LISP – device operating as both an ETR and ITR)

Y:\Training\Cisco Templates\Standard\Word Templates\New Brand Templates Standard Word_2017\copy block_NEW_template\legal blocks_new_110R-2017-1_updated.jpg

Comments
kadsteph
Cisco Employee
Cisco Employee
‎11-06-2023 01:03 PM

Comments on this article have been closed.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents