SDA Fabric Border Replacement Procedure

[vaisharm]

Introduction

This document describes typical steps needed to replace Fabric border(s) in an SDA Fabric with redundant Border Nodes. This procedure is for replacing the existing Border device(s) with an unlike device (different product/platform-id) (such as a newer, higher throughput platform) and is not an alternative like device replacement. for RMA workflow.  

Topology

The scenario covered below involves:

• Redundant (co-located) Fabric Border/Control-Plane nodes
• The existing physical links would be used to connect the new border.
• Both the borders were used as LAN automation seed devices for onboarding other devices such as Intermediate nodes(if used) and Fabric Edge switches. The steps below also cover scenario with manual underlay configuration.
• Both the borders are configured as the underlay and fabric/overlay Multicast RP
• Border-2 will be replaced first, followed by Border-1.

Pre-Work

• Upgrade the new/replacement devices to the desired IOS-XE version

Steps Involved

1. Disable the fabric multicast RP role - If the fabric borders are configured as overlay multicast RP in the fabric, we need to disable that role (one border device at a time) before these can be removed from the fabric. 
   
   NOTE: If overlay multicast is not configured or if the Fabric Borders are not configured as RP, skip this step and go directly to step 2.
   
   Here, both the Borders are configured as multicast RP in the fabric. As we are removing Border-2 first, we will start by removing the fabric/overlay multicast RP role from Border-2 by editing the fabric multicast configuration. 
   
   
   
   
   

   
   
   
   

   
   
   After the RP role is disabled for Border-2, the fabric infrastructure page would show the Border-2 icon without the RP role.
   

   

   
   
   
2. Remove Border-2 from the fabric.
   
   
   
   The image below indicates no fabric roles are now assigned to Border-2.
   

   

   
   
   
3. Delete Border-2 from Catalyst Center Inventory
   

   

   
   
   1. LAN Automation Scenario -  If the old Border-2 was used as a seed to onboard Edge/Intermediate node devices, the point-to-point interface configuration on the Edge/Intermediate node devices will be deleted by Catalyst Center when old Border-2 is deleted from Inventory. This would only affect the links which were configured by LAN automation between Border-2 and its directly connected neighbors.
   2. Manual Underlay - If the underlay was manually configured between Border-2 and the Edge/Intermediate node devices, make note of the point-to-point links and corresponding configuration so these can be restored on new Border-2.
      
      
4. Transition the physical links - After old Border-2 has been successfully removed from the fabric and inventory, move the physical link between old Border 2 and its directly connected neighbors (Fusion/Border-1/Edge/Intermediate Nodes) to the new Border-2 (we will use the hostname Border-2A for new Border-2 for the purpose of this document in the steps that follow). 
   1. Discover Border-2A under Catalyst Center (This would require Border-2A to have basic configuration so its reachable from Catalyst Center and is configured with at least CLI and SNMP credentials.)
   2. Provision Border-2A to the same site as old Border-2.
      

      

      
      
      
5. Configure underlay (connectivity with Fusion / Border-1 / Edge / Intermediate nodes)
   1. Manual Underlay -  If underlay was configured manually, re-configure it on the new border and use the same underlay config (IGP/IP addresses/multicast) as on old Border-2. 
   2. LAN Automation Scenario - If the Edge / Intermediate nodes were LAN automated originally, use the LAN Automation Add-link workflow to configure all the point-to-point links between Border-2A and its directly connected neighbors (Border-1 / Edge / Intermediate nodes).
      
      The new Border (Border-2A) would not be listed under the LAN automated devices. 
      

      

      
      
      For Border-2A to be listed for add link workflow, click on "Start LAN Automation" and initiate a LAN Automation session with both the borders selected (Primary/Secondary Seed device) and the link between the two borders.
      
      Select Border-1 as the primary seed device.
      

      

      
      
      Select Border-2A as the secondary seed device.
      

      

      
      
      Select the interface between Border-1 and Border-2A for LAN automation. Underlay multicast would be configured in step d. below.
      
      Stop LAN automation after seed device configuration is complete. This would configure the point-to-point link between the two borders with IP addresses and ISIS along with PIM (if Multicast was selected). 
      In case, you don't want to use this particular link, select any unused link on Border-2A alone and just start LAN automation and stop it after seed device configuration is complete. Nothing would be configured on the selected interface in this case but Border-2A would now be visible for add-link workflow.
      
      
      
      
      Add-Link Workflow:
      
      
      
      Select the point-to-point interface to be configured with Border-2A. Repeat the steps for each link.
      

      

      
      

      

      
      
      

   3. Confirm routing adjacencies with all directly connected neighbors
      
      

   4. If applicable, configure underlay multicast/RP/Loopback60000/MSDP configs as on original Border-2.  This step is needed even when LAN automation was used for underlay. 
      

      Border-2A (sample underlay multicast config):   

      ## PIM to be configured on Loopback0, Looopback60000 and all Point-to-Point links between the two Border nodes and the connected Edge devices (include any other interfaces as applicable/configured on original Border-2)

      ip pim sparse-mode       
      ip pim rp-address 192.168.208.34
      ip pim register-source Loopback0
      ip pim ssm default

       ## 192.168.208.37 is the Loopback0 on Border-1
      ip msdp peer 192.168.208.37 connect-source Loopback0
      ip msdp cache-sa-state
      ip msdp originator-id Loopback0
      interface Loopback60000
      ip address 192.168.208.34 255.255.255.255
      ip pim sparse-mode
      ip router isis
      
      

   5. Configure any manual CTS commands (if using white-list model) which were on the old Border-2.
      
      
      
6. Add Border-2A to the fabric 
   
   
   1. Add the new device to the fabric as a Border and/or Border/CP node (as applicable).
      

      
      
      

   2. Configure any L2 and L3 handoffs (just as on old Border-2). Make sure to use the same VLAN IDs as on original Border-2.
      

      
      Note: Update the L3 handoff IP addresses on the fusion device under BGP and handoff SVIs based on the new IP addresses assigned by Catalyst Center. 
      
      

   3. If SDA-Transit was configured on Border-2, re-configure it on Border-2A as well. 
      
   4. Manually configure per-VRF iBGP between the two border nodes (as on original Border-2). Not applicable for LISP PubSub fabric deployment.
   5. On Border-2A, confirm routing protocol adjacencies (BGP and other IGP as before) with fusion and Border-1.
   6. Add Border-2A as the Multicast RP from Fabric Multicast Configuration.
      (Topology view before enabling Overlay Multicast on Border-2A)
      
      
       Edit Multicast Configuration
      

      

      
      
      

      
      

      
      (Topology view after enabling Overlay Multicast on Border-2A)
      
      
      
      
      
7. Border-2A checks
   1. Check the routing table on Border-2A to confirm external routes  (and/or default route) are being received in appropriate VRFs. For LISP PubSub fabric deployment, make sure there is a default route in every VRF received from fusion.
   2. On the Edge nodes, confirm that CEF is pointing to the new border along with Border-1 as the next hop for external destinations (outside the fabric).
      
      show ip cef vrf <VRF_NAME> <External_Prefix> 
      
      Example:
      

      Edge-1#sh ip cef vrf DEFAULT_VN 4.2.2.2 
      0.0.0.0/0
        nexthop 192.168.208.36 LISP0.4098
        nexthop 192.168.208.37 LISP0.4098

      (Here, 192.168.208.36 is Border-2A while 192.168.208.37 is Border-1).

      
      
   3. The following steps are to ensure that Border-2A is handling all traffic as expected before we remove and replace Border-1 as well. 
      1. Make sure there is connectivity to Border-1 through console or management interface. 
      2. On Border-1, shutdown all links towards fusion, Border-2A, Edge or intermediate nodes.
      3. Confirm connectivity for fabric endpoints to external destinations through Border-2A
         
         
8. After all connectivity is successfully confirmed, un-shut all the links which were shutdown in steps 7(c)ii above so it is reachable from the Catalyst Center. If Catalyst Center can reach Border-1 via the management interface on Border-1 and that's the IP addresses used to manage Border-1, then this step (8) is not needed. Next, we will proceed with Border-1 replacement steps.
   
   
   
9. Remove fabric/overlay multicast RP role from Border-1 by editing the fabric multicast configuration.
   
   
   1. Remove Border-1 as an RP device
      

      

      
      
      
      
   2. Removing Border-1 may clear Border-2A entry as well on the GUI. Just select Border-2A again as the RP as below.
      

      

      
      
      
      
      
   3.   Summary page should show only Border-2A
      

      

      
      
      
      
10. Delete Border-1 from Catalyst Center Inventory (same as step 3 above).
    
    
11. Follow steps 4 thru 6 above (move physical links, discover, provision, underlay configuration, P2P link addition, underlay multicast/PIM, CTS, etc. ) for replacing Border-1 now with Border-1A and re-configuring it just as the old Border-1 device.
    

    
    
    

12. Perform traffic forwarding checks (as in step 7a, 7b above) to ensure fabric edge nodes point to both Border-2A and Border-1A for external destination. This step marks the successful replacement for both the fabric borders.