Seeking Guidance on SDA Deployment - Fusion FWs and BGP Configuration

[thenetadmin]

Hey Cisco Community,

Sorry for long post.

We are working on an SDA solution for a customer who plans to retain a pair of 3rd party firewalls as the fusion device. This is entirely greenfield deployment.

Network design involves connecting border nodes to the fusion firewall via BGP, using VLANs on trunk interfaces (dot1q trunk). I would appreciate some insights from the community on a few questions related to this setup:

1. We will deploy Catalyst 9600R as border nodes. Should we configure them as a Stackwise pair, or would it be better to set them up as two separate fabric borders?

2. If we proceed with separate borders rather than a StackWise setup, would we need to manually configure IBGP between the border nodes, or is there an option to automate this in a recent DNAC software release?

3. Given that the firewalls are in HA mode (active & standby), how would traffic flow be if the borders are configured in either Stackwise mode or as two separate nodes

4.  The customer has two distinct firewalls—one for the Internet Edge (WAN/internet traffic) and another for the Data Center (DC traffic). Should we establish two separate BGP peering—one with the Internet Edge firewall for internet-bound traffic and another with the DC firewall for data center-bound traffic? This would help ensure that internet traffic routes through the Internet Edge firewall and DC traffic routes through the DC firewall.

5. Should the Cisco WLC connect in the DC  or directly to the border nodes? Should be fabric enabled or not?

Thanks in advance.

[thenetadmin]

Hi @Andrii Oliinyk 

Thanks for all the feedback! Appreciate your time

• We’ve updated the design to use the data center distribution switch as the Fusion device for L3 handoff. The firewalls will also connect to this L3 switch, and we’ll assign the zones or interfaces on the firewalls to specific VRFs accordingly to advertise the routes.

• If we connect the WLC to the Fusion device, what’s the best approach to allow Fabric-enabled APs to reach the WLC.

• Additionally, if we utilize LISP Pub/Sub to achieve Border Redundancy, would a physical link between the BN/CP nodes required?

[Andrii Oliinyk]

1. sounds good
2. Approach for APs to reach WLC doesnt change. It's CAPWAP.
3. https://community.cisco.com/t5/software-defined-access-sd-access/sd-access-border-nodes-interconnection-design/td-p/5137340#:~:text=The%20CatC%20automated,of%20this%20thread.

[thenetadmin]

Hi @Andrii Oliinyk 

many thanks for your replies and appreciate your time and efforts. This discussion is getting interesting and learning new things.

1. Since we updated the design and introduced Nexus Switch as Fusion, what’s the recommended L3 handoff configuration now?. Firewalls HA are connected to the same Nexus switches as well.
2. For border redundancy, if the Fabric Site is Pub/Sub and BN/CP are combined on the same physical device, per-VRF (VPNv4) iBGP sessions are indeed handled automatically configured by the Catalyst Center. No Need for a Physical Link Between BN/CP Nodes?
3. What happens if BN1/CP1 loses connectivity with the Fusion switch or if BN1/CP1 fails completely? How does the convergence occur?

[jedolphi · Accepted solution]

Hi, haven't digested the whole thread, just answering the most recent question.

1. In a Pub/Sub network each BN should have at least one Layer 3 handoff (EBGP peering) to fusion device.

2. It's not strictly necessary but it is recommended to have at least one point-to-point routed underlay link between BN pairs. That link would use whatever routing protocol the rest of the underlay uses e.g. ISIS.

3. If BN1/CP1 loses connectivity to fusion it essentially removes itself  a valid path in LISP Pub/Sub, I will be covering this in my new Cisco Live Melbourne LISP talk next week, remind me in a few weeks and I'll share the recording link, if you're interested. If BN1/CP1 completely fails then it is withdrawn from underly IGP and that triggers all other nodes in the SDA fabric to route to BN2/CP2.

Best regards, Jerome

[Andrii Oliinyk]

Hi Jerom
with regard to automation of pair BN|CP interconnect, could u please clarify when VPNv4 BGP peering setup happens exactly?

tnx

[jedolphi]

Hi Andy, if you mean the underlay P2P routed interconnect between BNs then:

1. If LAN Automation was used to build the underlay then it can be added with LAN Automation.

2. If underlay was built manully then it will need to be added manually.

[Andrii Oliinyk]

Hi Jerom
Option 1: With LAN-automation DNAC only configures L3 IS-IS driven Underlay, not BGP. Couple of Qs here pls:
 a) if we want this BN|CP interconnect to be comprised with 2 physical links & configured by DNAC, we 1st need to discover 1st BN|CP w/o LAN-automation (bc if we do it from Fusion-switches as seed devices we will obtain L3 IS-IS driven interconnect(s) between BN|CP1 & FNs & with it we later wont be able to conduct L3-handoff on this uplinks from BN|CP1). & than we implement LAN-automation for 2nd BN|CP, right? & if we want this interconnect to be a L3-portchannel instead of 2 routed physical links, LAN-automation cannot help & therefore we need to fallback to manual BN|CP-interconnect configuration, right?
b) lets imagine somehow we've reached state when both BN|CPs are as such in the Fabric (meaning that we just assigned them BN|CP roles but didnt start any handoff), ok? how do we trigger configuring this VPNv4 BGP peering between them?  
thanks

[Andrii Oliinyk]

1. Recommended L3-handoff is one automated by DNAC via co-named workflow in BN|CP configuration pane.
2. u could conclude that w/o direct link bw BN|CPs & still BGP automated between BN|CP's Lo0 session would traverse EdgeNode layer. Does it sound good for u?
3. BN|CP1 either undeclares itself as PETR or disappear form Underlay & thus stops to participate in traffic transfer. Convergence occurs as usually - from lower to up layers. 

[thenetadmin]

Thank you @Andrii Oliinyk  @jedolphi 

Few more queries:

1- If Nexus are Fusion device and having eBGP peering with BN/CP, and the firewalls connect to the same fusion device as per the recent diagram ? What would be link configuration between Firewall and fusion? Does that link have to part of the VRFs? Firewalls need to advertise routes to VRFs for endpoints in fabric can reach data center resources.

2- In a Pub/Sub network each BN should have at least one Layer 3 handoff (EBGP peering) to fusion device. Does it required to be full mesh? If full mesh, do we need iBGP configure between two BN/CPs?

3-  When doing green field deployment, shall we add BN1/CP1 into DNAC, provision and do all underlay and overlay config then add BN2/CP2 and apply the config? Any recommended sequence

4-  Can we configure the port channel between two BN/CPs and automate the config by DNAC? And when does the configuration between two BN/CPs get pushed by DNAC? Please clarify.

[Andrii Oliinyk]

1. FN-FW connectivity design depends on many factors i dont know in your case. Generally it's totally up to u according to business needs & dependencies of your network. Just an example option for the case when all VRFs except Guest are using domestic proxy in DC to access FW: extend corporate VRFs to FW as is, use static routing for the traffic exchange between them, ensure proxy has access to the Inet-FW, extend Guest VRF to Inet-FW...  
2. It's recommended practice to have each BN|CP to peer with each FN. & though VPNv4 iBGP is declared to be  necessary in many sources, from my pov in case of redundant peering between BN|CPs & FNs former is not needed. Let @jedolphi to correct me if i'm wrong
3. My experience in this topic is not good for optimal reference as u can see me asking @jedolphi on similar topics. What will work form my pov in more or less simple case (assumingly u created in advance all necessary elements to provision your devices - Network Settings, L3,L2-VNs, Fabric site, network templates, wired & wireless profiles, auth templates etc): a) manually preconfigure both BN|CPs with IS-IS interconnect for discovery from DNAC. ensure VPNv4 iBGP peering between BN|CPs is in place, otherwise configure it manually; b) make LAN-automation for Edge nodes to have Underlay fully ready at the end; c)  provision BN|CPs with their roles & handoffs; d)  provision site WLC; e) provision EdgeNodes with their roles; f) onboard & provision APs. END. 
4. this is what i've been asking @jedolphi about. Currently we configure PC between BN|CPs manually as well as IS-IS on them. so in DNAC we only do Day-N provisioning, add switches as BN|CPs,  handoffs & that's it.
would appreciate @jedolphi to bring more clarity on gaps indicated in this thread.

[jedolphi · Accepted solution]

The practice of putting manual per-VRF IBGP between BNs was specific to LISP/BGP. New deployments should use LISP Pub/Sub please. LISP/BGP deployments should migrate to LISP Pub/Sub when we release the migration tool in future.

With Pub/Sub we no longer need any manual IBGP peering between BNs, please do not configure it, it won't help but it will waste your time and make the network more complex. Given we do not need IBGP between BNs there is no need to configure a port-channel between BNs. RECOMMENDED but not mandatory is p2p underlay routed link between BNs, at least one, maybe two. This enables outbound traffic to redirect from BN1 to BN2 quickly if BN1 loses connection to fusion.

[thenetadmin]

Thank you @jedolphi 

Gotcha thanks, since its new greenfield we will consider LISP Pub/Sub.

For P2P underlay routed link between BNs, is it automated by DNAC or manual. If manual to share the configuration. Can it be L3 port channel?

[jedolphi]

If you are using LAN Automation to bring up the SDA underlay, and if the two BNs were LAN Automation seeds, then the p2p routed links between BNs can be added by LAN Automation. There is a LAN Automation "Add Link" feature.

If the SDA underlay was configured manually then use the same IGP routing configuration as the rest of the SDA uderlay.

I did a video series on how to bring up SDA underlay and SDA fabric roles, it shows one way to accomplish it, see the LAN Automation videos in this video series.

[jedolphi]

Also, there is no strict need for port-channel in SDA underlay. You can add one manually if you wish, but port-channel adds complexity (more configuraiton, more protocols). SDA forwarding uses ECMP so two separate p2p routed links between the BNs is perfectly fine.

[thenetadmin]

Thank you @jedolphi 

Alright, so first, we will bring up the border nodes and provide them basic IP reachability with Catalyst Center. Add them to device inventory, provision and add to Fabric Site with LISP Pub/Sub. Then use LAN automation feature using these border nodes as seed devices, we will discover add the underlay switches into DNAC and lastly use the Add Link feature to create p2p underlay between Borders.

Hope I wrote the steps correctly?