Deploying FTDv virtualized security on SD-WAN Cat8K/UCS-E modules

[tokunath]

• Goal and tasks explained in this article
• Use Case benefits
• Solution architecture
• Hardware and Software
• Restrictions
• Prerequisites
• Solution Deployment
• High Level deployment workflow steps
• Detailed deployment instructions
• Task 1: Provision the UCS-E module in the Catalyst 8300
• Task 2: Install VMware ESXi Hypervisor on the UCS-E
• Task 3: Customize the ESXi network environment for the virtual firewall deployment
• Task 4: Deploy the FTDv Virtual Machine
• Task 5: Complete the Threat Defense Virtual Setup using the CLI
• Task 6: Configure high availability properties for the FTDv site pair from the Firepower Management Center (FMC)

 

Goal and tasks explained in this article 

Deploy Cisco Secure Firewall Threat Defense virtual security (FTDv) in routed, high availability mode on a pair of UCS E-Series server modules installed in Catalyst 8300 SD-WAN edge routers.   The following tasks are explained in this article.

1. Provision Catalyst 8300 WAN edge with UCS-E modules and provide CIMC management access via the shared backplane path
2. Install VMware ESXi and customize vSphere networking for the FTDv HA deployment
3. Deploy the FTDv installation package and configure device management parameters
4. Bring each FTDv instance into Firewall Management Center (FMC) control and configure into a High Availability (HA) pair
5. Deploy day-1 FTDv device configurations and security policies from the FMC

 

Use Case benefits 

This solution is for security sensitive customers deploying Cisco SD-WAN at critical business sites that connect to hybrid cloud networks in a variety of ways and require on-premises, full-stack security protection from potential threats coming in from local Internet connections or from other sites on the SD-WAN fabric.  Examples include large hospitals, logistics distribution centers, research and development sites, private or colocation data centers. These types of sites typically demand high bandwidth and have specific network connectivity and availability needs that are difficult to achieve with integrated features on a router performing cpu intensive tasks such as deep packet inspection, decryption, packet inspection and others that will lower forwarding throughput during peak usage times.

 

Solution architecture

This solution utilizes a pair of Catalyst 8300 SD-WAN edge routers with UCS-E server modules to deploy an integrated, high-throughput network and security solution that is resilient, feature rich, and fully protected from threats on the Internet or other sites on the SD-WAN fabric.  The logical network topology is shown in the image below.

The solution provides up to 10Gbps secure connectivity to the Internet and SD-WAN fabric by connecting the LAN, UCS-E, SD-WAN edge router and WAN components as shown in the image below

Hardware and Software

Item	Description	Version	Quantity
C8300-2N2S-4T2X	2RU WAN edge platform with 2 SM and 2 NIM slots plus 2 x 10Gbps and 4 x 1Gbps embedded Layer3 Ethernet ports	IOS-XE 17.6.3a	2
UCS-E1120D-M3/K9	Cisco UCS E1120D M3 double-wide Server blade, 12 core, 1.6 GHz Intel.    480GB SSD storage 64GB DRAM	UCSEM3_2.10 CIMC firmware 3.2(6.20180817145819 CPLD Version: 4.0 Hardware Version:2	2
VMware ESXi	License for VMware ESXi 6.7.0	Client version:1.33.4 Client build number:14093553 ESXi version:6.7.0 ESXi build number:15160138	2
FTDv50, 10Gbps	Firepower Threat Defense performance tier virtual license	Cisco Firepower Threat Defense for VMware v7.0.1 (build 84)	2
Firewall Management Center (FMC)	Firepower Management Center for VMWare	FMC for VMware 7.01 build 84	1

Restrictions

• The Catalyst 8300 routers support specific UCS-E server blades in service module slots. Refer to theCatalyst 8300 and UCS-E compatibility matrix for supported modules
• The backplane path between the UCS-E Series blade and router is not supported for virtual machine network data traffic. 
• The backplane path may be utilized for management traffic to/from the virtual machines, ESXi host, and CIMC controller. 

Prerequisites

• Catalyst 8300 WAN edge routers running in controller mode on IOS-XE 17.6.1 or newer 
• vManage/vBond/vSmart running SD-WAN 20.6.1 or newer 
• 4 x 10GE ports available on external LAN switch(es) for physical connectivity to the UCS-E and Catalyst 8300 router.
• Catalyst 8300 router management interface connected to external LAN switch.  (Management interface can connect to dedicated management VPN if available, or service VPN designated for management)
• Existing instance of Firewall Management Center (FMC) running 7.0.4 or newer reachable to/from each router management interface

 

Solution Deployment

This section provides detailed step-by-step instructions to manually deploy the solution from the various components. Automation of the solution is possible through scripting, but outside the scope of this article.

High Level deployment workflow steps

The high-level workflow to deploy the solution manually from the various components is shown below

Detailed deployment instructions

Task 1:  Provision the UCS-E module in the Catalyst 8300

This includes the Catalyst 8300 configurations to support the UCS-E server module, IP addressing and routing of the CIMC management path through the router/service module backplane path.

Tech Tip

Cisco Integrated Management controller (CIMC) is the management service for the UCS-E module.  IP management access to the CIMC can be configured through different internal or external interfaces on the UCS-E1120D server module  The diagram below illustrates how management access is gained to the UCS-E CIMC via the backplane path for this deployment. In this example, management traffic from service VPN 10 (GE0/0/0 interface) on the Catalyst 8300 is forwarded to the CIMC GE0 (console) interface via the ucse1/0/0 service module interface as shown in the diagram below. This is accomplished by assigning the internal GE0 interface with an available address from site LAN (Service VPN) space and configuring the internal ucse1/0/0 interface to share an IP address with a physical port connected to the LAN using an IP unnumbered configuration.  This method is also used to provide management access to the FTDv virtual machine and ESXi host in later steps.

Step1: Create vManage UCSE feature template that will be used to provision the Catalyst 8300 service module slot number and CIMC GE0 (Console) IPv4 address and default gateway.

Step2:  Create vManage template that will be used to provision the router UCSE interface IP address and static routes required to reach the CIMC, ESXi host and FTDV management host addresses.

Step3:  Ensure the “ARP Proxy” service is enabled in the Global Settings feature template

Step4:  Attach feature templates to Catalyst 8300 device templates and provision each router by supplying variables.

The IOS-XE cli configuration rendered on the Catalyst 8300  router is shown below.

cli configuration snippet

Step5:  Connect to the CIMC GUI interface and verify Server BIOS and CIMC firmware are at the latest versions.  UCSEM3_2.10 and 3.2(14.1) or newer recommended. Refer to the UCS-E Server module getting started guide for instructions on how to upgrade CIMC BIOS and UCS-E firmware

CIMC firmware

Task 2:  Install VMware ESXi Hypervisor on the UCS-E

The following example will install VMWare ESXi 7.0 from the CIMC KVM console and configure IP address on the management network Refer to the UCS-E getting started guide for detailed procedures on how to install the Hypervisor using the KVM console from CIMC

Before you begin

The VMware vSphere Hypervisor requires a customized image. To download the customized image, see Downloading the Customized VMware vSphere Hypervisor Image.

Task 2 step by step procedure

Step 1: Load the hypervisor image onto a workstation that can reach the CIMC.

Step 2: Log into the CIMC GUI, and from the top menu, click Launch KVM. The  Console opens in a separate window.

Step 3: From the KVM console, click the Virtual Media tab and click Activate Virtual Devices. Select accept this session and then click Apply

Step 4: Click the Virtual Media tab and click Map CD/DVD.

Step 5: Click Browse and select the hypervisor installation disk image. Click Open to mount the disk image, and then check the Mapped check box for the mounted disk image in the Virtual Media tab.

Step 6: Set the boot order to make the virtual CD/DVD drive as the boot device.

Step 7: Reboot the server. When the server reboots, it begins the installation process from the virtual CD/DVD drive. Refer to the platform installation guide for the installation process.

Step 8: Configure the Management network settings on the ESXi host.
Navigate to “Configure Management Network” from the KVM console of the CIMC. Select [x] vmnic2 adapter, which is mapped to the GE0 (console) interface of the UCS-E and the desired path for management. Add the IPv4 configuration details for the management network as shown below.

 

Use your browser to connect to the ESXi web client with https://<ESXi_IP_address>

Technical Tip – Understanding Network interface mapping on the UCS-E and VMWare ESXi Hypervisor

You can determine the port numbering of the E-Series Server by looking at the MAC addresses of the network interfaces. Note the following for the UCSE-1120D M3 series: The lowest numbered MAC address corresponds to the UCSE-1120D Server's GE0 (console) interface. The second lowest MAC address corresponds to the UCSE-1120D Server's GE1 interface. The third lowest MAC address corresponds to the UCSE-1120D Server's TE2 interface. The fourth lowest MAC address corresponds to the UCSE-1120D Server's TE3 interface. This is verified below by issuing the “show lom-mac-list” command from the CIMC command line interface, which can be accessed through ssh or by launching a console from the host Catalyst 8300 as shown below: CS3-8300-FTDv-1-1#hw-module session 1/0      Establishing session connect to subslot 1/0 To exit, type ^a^q E160S-FOC22512P1A /cimc/network # show lom-mac-list Interface                      MAC Address          ------------------------------ -------------------- Console                     00:f6:63:ba:18:d4    GE1                            00:f6:63:ba:18:d5    TE2                            00:f6:63:ba:18:d6    TE3                            00:f6:63:ba:18:d7 On Cisco UCS E Series M3 servers such as the UCSE-1120D-M3/K9, the VMware vSphere Hypervisor VMNIC interface ordering does not map to server’s lowest MAC address. After installing ESXi on M3 servers, the default vNIC interface ordering and server's NIC interface mappings are ordered as shown below. If desired, you can reconfigure ESXi to make VMNIC interface ordering to start with the server’s lowest MAC address by following the  procedures in the getting started guide:  UCS E Series M3 Servers:Reordering ESXi VMNIC Interface Number to Start with Server’s Lowest MAC Address For the purposes of this deployment, the default interface ordering was not reconfigured, and the resulting interface mapping is shown below: UCSE interface name ESXi adapter name MAC Address Comment GE0 (console) Vmnic2 00:f6:63:ba:18:d4    Management network GE1 Vmnic3 00:f6:63:ba:18:d5    Not used TE2 Vmnic0 00:f6:63:ba:18:d6    FTDV HA failover/sync TE3 Vmnic1 00:f6:63:ba:18:d7 FTDV Data

This mapping will be important in the next task where the ESXi network environment is prepared for the FTDV virtual machine deployment.

Task 3: Customize the ESXi network environment for the virtual firewall deployment

Before you begin

Understanding how virtual services integrate into network designs can be confusing unless there is a clear understanding the virtual and physical world mapping.   Before deployment it is recommended to develop a detailed diagram of the virtualized deployment as shown below

Task 3 step by step procedure

Step 1: Create two new virtual switches with uplinks to the external 10GE ports of the UCS-E.

Connect to the ESXi web client and navigate to Networking > Virtual switches > add standard virtual switch.

Create a new virtual switch named “vSwitch-TE3” with the following properties

• MTU – 1500
• Uplink1 – vmnic1 (maps to TE3 on UCSE)
• Security – Promiscuous mode, MAC address changes, forged transmits set to “accept” – required for Firewall VMs

Create a second virtual switch named “vSwitch-TE2” with the following properties

• MTU – 1500
• Uplink1 – vmnic0 (maps to TE2 on UCSE)
• Security – Promiscuous mode, MAC address changes, forged transmits set to “accept” – required for Firewall VMs such as FTDv

Step 2: Create new port groups “Data” and “HA”, specifying attributes for vSwitch access-link connectivity

Note: Port group “data” represents the VM access-links to virtual switch vSwitch-TE3. This port group will need to support trunking since the inside and outside interfaces of the FTDv are defined as sub-interfaces on the same physical network adapter.  This is done by configuring “4095” in the VLAN field which enables trunking support for any VLAN, as shown below. 

Port Group HA does not require trunking and can be left with the default VLAN ID setting of 0

The ESXi environment has been prepared for deployment of the FTDv Virtual Machine

Task 4: Deploy the FTDv Virtual Machine

Before you begin

Review the Cisco Secure Firewall Threat Defense Virtual Getting Started Guide chapter Deploy the Threat Defense Virtual on VMware to understand the feature support, guidelines, limitations and known issues.

The workflow to deploy the FTDv in VMware ESXi is shown below

Task 4 step by step procedure

Step 1: Download the threat defense virtual install package for VMware ESXi from Cisco.com, and save it to your local management computer: Package used for this guide: Cisco_Firepower_Threat_Defense_Virtual-7.0.4-55.tar.gz

Step 2: Unpack the tar file into a working directory. Do not remove any files from the directory. 

Step 3: In a browser, enter the ESXi target host name or IP address using the format http://host-name/ui or http://host-IP-address/ui.

Step 4: Right-click on Host in the VMware Host Client inventory and select Create/Register VM. The New Virtual Machine wizard opens.

Step 5: On the Select creation type page of the wizard, select Deploy a virtual machine from an OVF or OVA file and click Next.

Step 6: On the Select OVF and VMDK files page of the wizard:
a. Enter a name for your threat defense virtual machine.
b. Click the blue pane, browse to the directory where you unpacked the threat defense virtual tar file, and choose the ESXi OVF template and the accompanying VMDK file:

Step 7: Click Next.Your local system storage opens.

Step 8: Choose a datastore from the list of accessible datastores on the Select storage page of the wizard and click Next

Step 9: Configure the Deployment options that come packaged with the ESXi OVF for the threat defense virtual:
a. Network Mapping—Map the networks specified in the OVF template to networks in your inventory, and then select Next.
This design will require 4 network interfaces.
1. Management0-0 to the VM Network port group
2. Diagnostic to the VM Network port group
3. GigabitEthernet0-0 to the Data port group
4. GigabitEthernet0-1 to the HA port group
b. Deployment Type – Specify the number of Cores and Memory allocated. For deployments requiring up to 10Gbps throughput, select the maximum 16Core / 32GB option.
c. Disk provisioning—Select Thick format to store virtual machine virtual disks.
d. Power on automatically – Select this option

Step 10: 

On the Ready to complete page of the New virtual machine wizard, review the configuration settings for the virtual machine.
a. (Optional) Click Back to go back and review or modify the wizard settings.
b. (Optional) Click Cancel to discard the creation task and close the wizard.
c. Click Finish to complete the creation task and close the wizard.

After you complete the wizard, the ESXi host processes the VM; you can see the deployment status in the Recent Tasks pane. A successful deployment shows Completed successfully under the Results column.

The new threat defense virtual virtual machine instance then appears under the Virtual Machines inventory of the ESXi host. Booting up the new virtual machine could take up to 30 minutes.
To successfully register the threat defense virtual with the Cisco Licensing Authority, the threat defense virtual requires Internet access. You might need to perform additional configuration after deployment to achieve Internet access and successful license registration.

Task 5: Complete the Threat Defense Virtual Setup using the CLI

When you first log into a newly configured device, you must read and accept the EULA. Then, follow the setup prompts to change the administrator password, and configure the device’s network settings and firewall mode. Since this deployment will be configured and managed with Cisco Firepower Management Center, an additional CLI command will be used to bring it under management

Task 5 step by step procedure

Step 1: Open the VMware console.

Step 2: At the firepower login prompt, log in with the default credentials of username admin and the password Admin123

Step 3: When the threat defense virtual system boots, a setup wizard prompts you for the following information required to configure the system:
• Accept EULA and create a new admin password
• Define management port details including IPv4 address, subnet mask, default gateway, DNS, and if applicable HTTP proxy
• Assign system name (Each FTDv in an HA pair must have unique system name)
• Specify whether Management mode is "local", which uses the onboard device manager.  In this deployment we will answer "no" since we will be using the Firewall Management Console (FMC) to centrally manage FTDv devices. 

Step 4: Review the Setup wizard settings. Defaults or previously entered values appear in brackets. To accept previously entered values, press Enter

Step 5: Complete the system configuration and verify the settings.

Step 6: Add the FTDv to the Firepower Management Center issuing the “configure manager add <FMC IP address> <key>

At this point the FTDv appliance is in FMC managed mode and will not accept local configuration changes.  

Step 7:  Repeat steps 1-6 for the other FTDv instance in the second Cat8K-hosted UCS-E module 

Task 6: Configure high availability properties for the FTDv site pair from the Firepower Management Center (FMC)

Tech Tip : Note on FTDv High Availability (HA)

FTDv HA ensures site connectivity if an active FTDv device or link fails. With active/standby high availability, two devices are linked, so that if the active device fails, the standby device takes over and users should see no more than a brief connectivity problem.  All configurations on the selected primary unit are replicated to the selected secondary FTD unit. This includes the IP addresses on all interfaces except for management interface which must be unique. In order to create an HA between 2 FTD devices, the following conditions must be met. • Same FTD model, same version, same number and type of interfaces. • Both devices as part of the same group/domain in FMC • Have identical Network Time Protocol (NTP) configuration • Be fully deployed on the FMC without uncommitted changes • Be in the same firewall mode: routed or transparent. • Different hostname (Fully Qualified Domain Name (FQDN)) for both chassis. In order to check the chassis hostname, navigate to FTD CLI and run “show chassis detail”. If both fhave the same name, change the name of one, commit and unregister from FMC. Then re-register to FMC in order to proceed with the HA pair creation.

The FTDv High Availability deployment details for this deployment is illustrated in the diagram below

Task 6 step by step procedure

Step 1: Add both devices to the Firepower Management Center according to Add a Device to the FMC.
Note: Devices in this deployment have already been added to FMC from each FTDv cli in task 5 of this guide.

Step 2: Choose Devices > Device Management.

Step 3: From the Add drop-down menu, choose High Availability.

Step 4: Enter a display Name for the high availability pair.

Step 5: Under Device Type, choose Firepower Threat Defense.

Step 6: Choose the Primary Peer device for the high availability pair.

Step 7: Choose the Secondary Peer device for the high availability pair. Click Continue.

Step 8: Under LAN Failover Link, choose an Interface with enough bandwidth to reserve for failover communications.
Note This deployment uses the G0/1 interface of each FTDv instance which maps to the corresponding "TE3" interface on the UCS-E module

Step 9: Type any identifying Logical Name. 

Step 10: Type a Primary IPv4 address for the failover link on the active unit. (Optionally provide an IPv6 address)

Step 11: Type a Secondary IP address for the failover link on the standby unit. This IP address must be in the same subnet as the primary IP address. 

Step 12: If IPv4 addresses are used, type a Subnet Mask that applies to both the primary and secondary IP addresses. 

Step 13: Optionally, under Stateful Failover Link, choose the same Interface, or choose a different interface and enter the high availability configuration information.
Note 169.254.0.0/16 and fd00:0:0:*::/64 are internally used subnets and cannot be used for the failover or state links.

In this example the same interface (G0/1) is used for high availability and stateful failover between FTDv devices

Step 14: Optionally, choose Enabled and choose the Key Generation method for IPsec Encryption between the failover links. 

Step 15: Click OK. This process takes a few minutes as the process synchronizes system data.

Note on HA: Once devices are added in an HA pair, it is no longer possible to edit FTDv device configurations separately. Instead, configure interface parameters on the HA device that was created

Step 16: Configure the Data Interfaces (named inside and outside) on the HA pair.

Note: This design utilizes sub-interfaces under the G0/0 parent interface of each FTDv for the “inside” and “outside” data interfaces that will connect to the LAN switch 10G trunk interface. This was chosen since the G0/0 interface maps to the other 10GigE interface (TE2) on the UCS-E.

Step 16.1:  Configure Physical/Parent interface.  

From the FMC devices tab, select device management > “Branch101-HA“ > GigabitEthernet0/0 > Edit physical interface.

Enter name “parent_interface”, and enable the interface.  Ensure Mode is left to “None” and MTU "1500". No IP address is assigned to this parent interface, as they will be defined on subinterfaces created in the subsequent steps below

Step 16.2:  Configure logical "inside" and "outside" subinterfaces and assign primary IP addresses.  

Select “Add interfaces” to create sub-interfaces G0/0.1107 and G0/0.2107 adding parameters to the General and IPv4 tabs as shown in the images below.

Step 16.3. Configure standby IP address for the inside and outside interfaces.

Navigate to Branch101-HA > Monitored interfaces > Click pencil icon for outside and inside. Assign Standby IP addresses for both interfaces as shown in the below images

Click OK to return to the HA screen and review the configurations

Save the configuration settings and "Deploy" the changes to both FTDv devices

Step 17: Configure static routing with host tracking

Tech Tip on routed mode FTDv deployments

To route traffic to a non-connected host or network, you must define a route to the host or network, either using static or dynamic routing. Generally, you must configure at least one static route: a default route for all traffic that is not routed by other means to a default network gateway, typically the next hop router. One of the problems with static routes is that there is no inherent mechanism for determining if the route is up or down. They remain in the routing table even if the next hop gateway becomes unavailable, which can cause traffic blackholing. Static routes are only removed from the routing table if the associated interface on the FTD device goes down. This can be especially challenging in virtualized deployments where FTDv link state does not drop in deployments where the virtualization server interface connects to the next-hop gateways indirectly via a LAN switch. The FTD device implements static route tracking by associating a static route with a monitoring target host on the destination network that the FTD device monitors using ICMP echo requests. If an echo reply is not received within a specified time period, the host is considered down, and the associated route is removed from the routing table. An untracked backup route with a higher metric is used in place of the removed route. When selecting a monitoring target, you need to make sure that it can respond to ICMP echo requests. The target can be any network object that you choose, but you should consider using a server beyond the next-hop gateway to ensure WAN connectivity is available.

This deployment utilizes static default routes on the FTDv HA pair, with the SD-WAN Edge routers, BR101-WAN1 and BR101-WAN2 serving as the next-hop gateways. Static route tracking of Cisco Umbrella DNS addresses 208.67.222.222 and 208.67.220.220 is enabled to ensure the next-hop gateways can reach these hosts over the Internet WAN transport path.  The SD-WAN edge routers similarly track DIA static default routes to these same Umbrella DNS addresses

To access the static route configuration on FMC, navigate to device management > Devices > Branch101-HA > Routing > Static Route > + Add Route > 

Select “IPv4” button as Type, “outside” for Interface, choose and Add the “any-ipv4” object to the Selected Network pane.
Select “+’ at the Gateway dropdown to add new objects “WAN-Edge-Router1” and “WAN-Edge-Router2” and supply the IPv4 addresses associated with the Gig0/0/2 (service VPN 1) interfaces of each router.

Select WAN-Edge-Router1 as the first Gateway with a Metric of “1” and select “+” next to Route Tracking to add the Umbrella DNS IPv4 address and add the details as shown in the images below.

Save and select the SLA monitor object UmbrellaDNS1 for each of the static route configurations

Save and deploy the configuration.

Step 18: Verify High Availability

Refer to the troubleshooting technote Configure FTD High Availability on Firepower Appliances for detailed procedures to verify and operate High Availability.  This includes steps to verify HA configurations and license, and procedures to switch the failure roles, break, disable or suspend the HA pair.

Step 19:  Deploy FTDv security policies

Security policy configuration on the FTDv HA pair is out of scope for this guide, as every deployment is unique.  Refer to the configuration guide Firepower Management Center Configuration Guide, Version 7.0 for procedures on how to create Access Control, Advanced Malware Protection (AMP) and File Control, TID Intelligence and Threat Analysis, Intrusion Detection and Prevention, Advanced Network Analysis and Preprocessing, Discovery and Identity, Correlation and Compliance, Reporting and Alerting.