Deploy Cisco Secure Firewall Threat Defense virtual security (FTDv) in routed, high availability mode on a pair of UCS E-Series server modules installed in Catalyst 8300 SD-WAN edge routers. The following tasks are explained in this article.
This solution is for security sensitive customers deploying Cisco SD-WAN at critical business sites that connect to hybrid cloud networks in a variety of ways and require on-premises, full-stack security protection from potential threats coming in from local Internet connections or from other sites on the SD-WAN fabric. Examples include large hospitals, logistics distribution centers, research and development sites, private or colocation data centers. These types of sites typically demand high bandwidth and have specific network connectivity and availability needs that are difficult to achieve with integrated features on a router performing cpu intensive tasks such as deep packet inspection, decryption, packet inspection and others that will lower forwarding throughput during peak usage times.
This solution utilizes a pair of Catalyst 8300 SD-WAN edge routers with UCS-E server modules to deploy an integrated, high-throughput network and security solution that is resilient, feature rich, and fully protected from threats on the Internet or other sites on the SD-WAN fabric. The logical network topology is shown in the image below.
The solution provides up to 10Gbps secure connectivity to the Internet and SD-WAN fabric by connecting the LAN, UCS-E, SD-WAN edge router and WAN components as shown in the image below
2RU WAN edge platform with 2 SM and 2 NIM slots plus 2 x 10Gbps and 4 x 1Gbps embedded Layer3 Ethernet ports
Cisco UCS E1120D M3 double-wide Server blade, 12 core, 1.6 GHz Intel.
480GB SSD storage64GB DRAM
CIMC firmware 3.2(6.20180817145819
CPLD Version: 4.0
License for VMware ESXi 6.7.0
Client build number:14093553
ESXi version:6.7.0ESXi build number:15160138
Firepower Threat Defense performance tier virtual license
Cisco Firepower Threat Defense for VMware v7.0.1 (build 84)
Firewall Management Center (FMC)
Firepower Management Center for VMWare
FMC for VMware 7.01 build 84
This section provides detailed step-by-step instructions to manually deploy the solution from the various components. Automation of the solution is possible through scripting, but outside the scope of this article.
The high-level workflow to deploy the solution manually from the various components is shown below
This includes the Catalyst 8300 configurations to support the UCS-E server module, IP addressing and routing of the CIMC management path through the router/service module backplane path.
Cisco Integrated Management controller (CIMC) is the management service for the UCS-E module. IP management access to the CIMC can be configured through different internal or external interfaces on the UCS-E1120D server module
The diagram below illustrates how management access is gained to the UCS-E CIMC via the backplane path for this deployment. In this example, management traffic from service VPN 10 (GE0/0/0 interface) on the Catalyst 8300 is forwarded to the CIMC GE0 (console) interface via the ucse1/0/0 service module interface as shown in the diagram below.
This is accomplished by assigning the internal GE0 interface with an available address from site LAN (Service VPN) space and configuring the internal ucse1/0/0 interface to share an IP address with a physical port connected to the LAN using an IP unnumbered configuration. This method is also used to provide management access to the FTDv virtual machine and ESXi host in later steps.
Step1: Create vManage UCSE feature template that will be used to provision the Catalyst 8300 service module slot number and CIMC GE0 (Console) IPv4 address and default gateway.
Step2: Create vManage template that will be used to provision the router UCSE interface IP address and static routes required to reach the CIMC, ESXi host and FTDV management host addresses.
Step3: Ensure the “ARP Proxy” service is enabled in the Global Settings feature template
Step4: Attach feature templates to Catalyst 8300 device templates and provision each router by supplying variables.
The IOS-XE cli configuration rendered on the Catalyst 8300 router is shown below.
Step5: Connect to the CIMC GUI interface and verify Server BIOS and CIMC firmware are at the latest versions. UCSEM3_2.10 and 3.2(14.1) or newer recommended. Refer to the UCS-E Server module getting started guide for instructions on how to upgrade CIMC BIOS and UCS-E firmware
The following example will install VMWare ESXi 7.0 from the CIMC KVM console and configure IP address on the management network Refer to the UCS-E getting started guide for detailed procedures on how to install the Hypervisor using the KVM console from CIMC
Before you begin
The VMware vSphere Hypervisor requires a customized image. To download the customized image, see Downloading the Customized VMware vSphere Hypervisor Image.
Task 2 step by step procedure
Step 1: Load the hypervisor image onto a workstation that can reach the CIMC.
Step 2: Log into the CIMC GUI, and from the top menu, click Launch KVM. The Console opens in a separate window.
Step 3: From the KVM console, click the Virtual Media tab and click Activate Virtual Devices. Select accept this session and then click Apply
Step 4: Click the Virtual Media tab and click Map CD/DVD.
Step 5: Click Browse and select the hypervisor installation disk image. Click Open to mount the disk image, and then check the Mapped check box for the mounted disk image in the Virtual Media tab.
Step 6: Set the boot order to make the virtual CD/DVD drive as the boot device.
Step 7: Reboot the server. When the server reboots, it begins the installation process from the virtual CD/DVD drive. Refer to the platform installation guide for the installation process.
Step 8: Configure the Management network settings on the ESXi host.
Navigate to “Configure Management Network” from the KVM console of the CIMC. Select [x] vmnic2 adapter, which is mapped to the GE0 (console) interface of the UCS-E and the desired path for management. Add the IPv4 configuration details for the management network as shown below.
Use your browser to connect to the ESXi web client with https://<ESXi_IP_address>
Technical Tip – Understanding Network interface mapping on the UCS-E and VMWare ESXi Hypervisor
You can determine the port numbering of the E-Series Server by looking at the MAC addresses of the network interfaces. Note the following for the UCSE-1120D M3 series:
This is verified below by issuing the “show lom-mac-list” command from the CIMC command line interface, which can be accessed through ssh or by launching a console from the host Catalyst 8300 as shown below:
CS3-8300-FTDv-1-1#hw-module session 1/0
Establishing session connect to subslot 1/0
To exit, type ^a^q
E160S-FOC22512P1A /cimc/network # show lom-mac-list
Interface MAC Address
On Cisco UCS E Series M3 servers such as the UCSE-1120D-M3/K9, the VMware vSphere Hypervisor VMNIC interface ordering does not map to server’s lowest MAC address. After installing ESXi on M3 servers, the default vNIC interface ordering and server's NIC interface mappings are ordered as shown below.
If desired, you can reconfigure ESXi to make VMNIC interface ordering to start with the server’s lowest MAC address by following the procedures in the getting started guide: UCS E Series M3 Servers:Reordering ESXi VMNIC Interface Number to Start with Server’s Lowest MAC Address
For the purposes of this deployment, the default interface ordering was not reconfigured, and the resulting interface mapping is shown below:
This mapping will be important in the next task where the ESXi network environment is prepared for the FTDV virtual machine deployment.
Before you begin
Understanding how virtual services integrate into network designs can be confusing unless there is a clear understanding the virtual and physical world mapping. Before deployment it is recommended to develop a detailed diagram of the virtualized deployment as shown below
Task 3 step by step procedure
Step 1: Create two new virtual switches with uplinks to the external 10GE ports of the UCS-E.
Connect to the ESXi web client and navigate to Networking > Virtual switches > add standard virtual switch.
Create a new virtual switch named “vSwitch-TE3” with the following properties
Create a second virtual switch named “vSwitch-TE2” with the following properties
Step 2: Create new port groups “Data” and “HA”, specifying attributes for vSwitch access-link connectivity
Note: Port group “data” represents the VM access-links to virtual switch vSwitch-TE3. This port group will need to support trunking since the inside and outside interfaces of the FTDv are defined as sub-interfaces on the same physical network adapter. This is done by configuring “4095” in the VLAN field which enables trunking support for any VLAN, as shown below.
Port Group HA does not require trunking and can be left with the default VLAN ID setting of 0
The ESXi environment has been prepared for deployment of the FTDv Virtual Machine
Before you begin
Review the Cisco Secure Firewall Threat Defense Virtual Getting Started Guide chapter Deploy the Threat Defense Virtual on VMware to understand the feature support, guidelines, limitations and known issues.
The workflow to deploy the FTDv in VMware ESXi is shown below
Task 4 step by step procedure
Step 1: Download the threat defense virtual install package for VMware ESXi from Cisco.com, and save it to your local management computer: Package used for this guide: Cisco_Firepower_Threat_Defense_Virtual-7.0.4-55.tar.gz
Step 2: Unpack the tar file into a working directory. Do not remove any files from the directory.
Step 4: Right-click on Host in the VMware Host Client inventory and select Create/Register VM. The New Virtual Machine wizard opens.
Step 5: On the Select creation type page of the wizard, select Deploy a virtual machine from an OVF or OVA file and click Next.
Step 6: On the Select OVF and VMDK files page of the wizard:
a. Enter a name for your threat defense virtual machine.
b. Click the blue pane, browse to the directory where you unpacked the threat defense virtual tar file, and choose the ESXi OVF template and the accompanying VMDK file:
Step 7: Click Next.Your local system storage opens.
Step 8: Choose a datastore from the list of accessible datastores on the Select storage page of the wizard and click Next
Step 9: Configure the Deployment options that come packaged with the ESXi OVF for the threat defense virtual:
a. Network Mapping—Map the networks specified in the OVF template to networks in your inventory, and then select Next.
This design will require 4 network interfaces.
1. Management0-0 to the VM Network port group
2. Diagnostic to the VM Network port group
3. GigabitEthernet0-0 to the Data port group
4. GigabitEthernet0-1 to the HA port group
b. Deployment Type – Specify the number of Cores and Memory allocated. For deployments requiring up to 10Gbps throughput, select the maximum 16Core / 32GB option.
c. Disk provisioning—Select Thick format to store virtual machine virtual disks.
d. Power on automatically – Select this option
On the Ready to complete page of the New virtual machine wizard, review the configuration settings for the virtual machine.
a. (Optional) Click Back to go back and review or modify the wizard settings.
b. (Optional) Click Cancel to discard the creation task and close the wizard.
c. Click Finish to complete the creation task and close the wizard.
After you complete the wizard, the ESXi host processes the VM; you can see the deployment status in the Recent Tasks pane. A successful deployment shows Completed successfully under the Results column.
The new threat defense virtual virtual machine instance then appears under the Virtual Machines inventory of the ESXi host. Booting up the new virtual machine could take up to 30 minutes.
To successfully register the threat defense virtual with the Cisco Licensing Authority, the threat defense virtual requires Internet access. You might need to perform additional configuration after deployment to achieve Internet access and successful license registration.
When you first log into a newly configured device, you must read and accept the EULA. Then, follow the setup prompts to change the administrator password, and configure the device’s network settings and firewall mode. Since this deployment will be configured and managed with Cisco Firepower Management Center, an additional CLI command will be used to bring it under management
Task 5 step by step procedure
Step 1: Open the VMware console.
Step 2: At the firepower login prompt, log in with the default credentials of username admin and the password Admin123
Step 3: When the threat defense virtual system boots, a setup wizard prompts you for the following information required to configure the system:
• Accept EULA and create a new admin password
• Define management port details including IPv4 address, subnet mask, default gateway, DNS, and if applicable HTTP proxy
• Assign system name (Each FTDv in an HA pair must have unique system name)
• Specify whether Management mode is "local", which uses the onboard device manager. In this deployment we will answer "no" since we will be using the Firewall Management Console (FMC) to centrally manage FTDv devices.
Step 4: Review the Setup wizard settings. Defaults or previously entered values appear in brackets. To accept previously entered values, press Enter
Step 5: Complete the system configuration and verify the settings.
Step 6: Add the FTDv to the Firepower Management Center issuing the “configure manager add <FMC IP address> <key>
At this point the FTDv appliance is in FMC managed mode and will not accept local configuration changes.
Step 7: Repeat steps 1-6 for the other FTDv instance in the second Cat8K-hosted UCS-E module
|Tech Tip : Note on FTDv High Availability (HA)|
|FTDv HA ensures site connectivity if an active FTDv device or link fails. With active/standby high availability, two devices are linked, so that if the active device fails, the standby device takes over and users should see no more than a brief connectivity problem. All configurations on the selected primary unit are replicated to the selected secondary FTD unit. This includes the IP addresses on all interfaces except for management interface which must be unique.
In order to create an HA between 2 FTD devices, the following conditions must be met.
• Same FTD model, same version, same number and type of interfaces.
• Both devices as part of the same group/domain in FMC
• Have identical Network Time Protocol (NTP) configuration
• Be fully deployed on the FMC without uncommitted changes
• Be in the same firewall mode: routed or transparent.
• Different hostname (Fully Qualified Domain Name (FQDN)) for both chassis. In order to check the chassis hostname, navigate to FTD CLI and run “show chassis detail”. If both fhave the same name, change the name of one, commit and unregister from FMC. Then re-register to FMC in order to proceed with the HA pair creation.
The FTDv High Availability deployment details for this deployment is illustrated in the diagram below
Task 6 step by step procedure
Step 1: Add both devices to the Firepower Management Center according to Add a Device to the FMC.
Note: Devices in this deployment have already been added to FMC from each FTDv cli in task 5 of this guide.
Step 2: Choose Devices > Device Management.
Step 3: From the Add drop-down menu, choose High Availability.
Step 4: Enter a display Name for the high availability pair.
Step 5: Under Device Type, choose Firepower Threat Defense.
Step 6: Choose the Primary Peer device for the high availability pair.
Step 7: Choose the Secondary Peer device for the high availability pair. Click Continue.
Step 8: Under LAN Failover Link, choose an Interface with enough bandwidth to reserve for failover communications.
Note This deployment uses the G0/1 interface of each FTDv instance which maps to the corresponding "TE3" interface on the UCS-E module
Step 9: Type any identifying Logical Name.
Step 10: Type a Primary IPv4 address for the failover link on the active unit. (Optionally provide an IPv6 address)
Step 11: Type a Secondary IP address for the failover link on the standby unit. This IP address must be in the same subnet as the primary IP address.
Step 12: If IPv4 addresses are used, type a Subnet Mask that applies to both the primary and secondary IP addresses.
Step 13: Optionally, under Stateful Failover Link, choose the same Interface, or choose a different interface and enter the high availability configuration information.
Note 169.254.0.0/16 and fd00:0:0:*::/64 are internally used subnets and cannot be used for the failover or state links.
In this example the same interface (G0/1) is used for high availability and stateful failover between FTDv devices
Step 14: Optionally, choose Enabled and choose the Key Generation method for IPsec Encryption between the failover links.
Step 15: Click OK. This process takes a few minutes as the process synchronizes system data.
Note on HA: Once devices are added in an HA pair, it is no longer possible to edit FTDv device configurations separately. Instead, configure interface parameters on the HA device that was created
Step 16: Configure the Data Interfaces (named inside and outside) on the HA pair.
Note: This design utilizes sub-interfaces under the G0/0 parent interface of each FTDv for the “inside” and “outside” data interfaces that will connect to the LAN switch 10G trunk interface. This was chosen since the G0/0 interface maps to the other 10GigE interface (TE2) on the UCS-E.
Step 16.1: Configure Physical/Parent interface.
From the FMC devices tab, select device management > “Branch101-HA“ > GigabitEthernet0/0 > Edit physical interface.
Enter name “parent_interface”, and enable the interface. Ensure Mode is left to “None” and MTU "1500". No IP address is assigned to this parent interface, as they will be defined on subinterfaces created in the subsequent steps below
Step 16.2: Configure logical "inside" and "outside" subinterfaces and assign primary IP addresses.
Select “Add interfaces” to create sub-interfaces G0/0.1107 and G0/0.2107 adding parameters to the General and IPv4 tabs as shown in the images below.
Step 16.3. Configure standby IP address for the inside and outside interfaces.
Navigate to Branch101-HA > Monitored interfaces > Click pencil icon for outside and inside. Assign Standby IP addresses for both interfaces as shown in the below images
Click OK to return to the HA screen and review the configurations
Save the configuration settings and "Deploy" the changes to both FTDv devices
Step 17: Configure static routing with host tracking
|Tech Tip on routed mode FTDv deployments|
To route traffic to a non-connected host or network, you must define a route to the host or network, either using static or dynamic routing. Generally, you must configure at least one static route: a default route for all traffic that is not routed by other means to a default network gateway, typically the next hop router.
One of the problems with static routes is that there is no inherent mechanism for determining if the route is up or down. They remain in the routing table even if the next hop gateway becomes unavailable, which can cause traffic blackholing. Static routes are only removed from the routing table if the associated interface on the FTD device goes down. This can be especially challenging in virtualized deployments where FTDv link state does not drop in deployments where the virtualization server interface connects to the next-hop gateways indirectly via a LAN switch.
The FTD device implements static route tracking by associating a static route with a monitoring target host on the destination network that the FTD device monitors using ICMP echo requests. If an echo reply is not received within a specified time period, the host is considered down, and the associated route is removed from the routing table. An untracked backup route with a higher metric is used in place of the removed route. When selecting a monitoring target, you need to make sure that it can respond to ICMP echo requests. The target can be any network object that you choose, but you should consider using a server beyond the next-hop gateway to ensure WAN connectivity is available.
This deployment utilizes static default routes on the FTDv HA pair, with the SD-WAN Edge routers, BR101-WAN1 and BR101-WAN2 serving as the next-hop gateways. Static route tracking of Cisco Umbrella DNS addresses 126.96.36.199 and 188.8.131.52 is enabled to ensure the next-hop gateways can reach these hosts over the Internet WAN transport path. The SD-WAN edge routers similarly track DIA static default routes to these same Umbrella DNS addresses
To access the static route configuration on FMC, navigate to device management > Devices > Branch101-HA > Routing > Static Route > + Add Route >
Select “IPv4” button as Type, “outside” for Interface, choose and Add the “any-ipv4” object to the Selected Network pane.
Select “+’ at the Gateway dropdown to add new objects “WAN-Edge-Router1” and “WAN-Edge-Router2” and supply the IPv4 addresses associated with the Gig0/0/2 (service VPN 1) interfaces of each router.
Select WAN-Edge-Router1 as the first Gateway with a Metric of “1” and select “+” next to Route Tracking to add the Umbrella DNS IPv4 address and add the details as shown in the images below.
Save and select the SLA monitor object UmbrellaDNS1 for each of the static route configurations
Save and deploy the configuration.
Step 18: Verify High Availability
Refer to the troubleshooting technote Configure FTD High Availability on Firepower Appliances for detailed procedures to verify and operate High Availability. This includes steps to verify HA configurations and license, and procedures to switch the failure roles, break, disable or suspend the HA pair.
Step 19: Deploy FTDv security policies
Security policy configuration on the FTDv HA pair is out of scope for this guide, as every deployment is unique. Refer to the configuration guide Firepower Management Center Configuration Guide, Version 7.0 for procedures on how to create Access Control, Advanced Malware Protection (AMP) and File Control, TID Intelligence and Threat Analysis, Intrusion Detection and Prevention, Advanced Network Analysis and Preprocessing, Discovery and Identity, Correlation and Compliance, Reporting and Alerting.