Cisco Defense Orchestrator (CDO) Onboarding HA Pair of FTD Devices - Needs help

[bmurphree@myemma.com]

Does anyone have a tested and successful procedure to onboard an HA pair of FTD devices (already set as active/standby via FDM)?

I've been in a pilot program of CDO for several months (the first few spent fruitlessly gaining access to CDO).

I've been testing a scenario where I want to onboard an already configured HA pair of FTD devices.

I have tried quite a number of different iterations of methods, none of which have been successful, least of which is Cisco's own procedure (https://docs.defenseorchestrator.com/Configuration_Guides/Onboard_Devices_and_Services/0021_Onboard_a_Firepower_Threat_Defense_Device_High_Availability_Pair). In fact Cisco's procedure for this doesn't even begin to work.

Using the procedure below, is the closest to onboarding the pair that I have gotten, but it still fails:

1. De-Register both devices (license registration or Eval MUST match!)

2. Disable any other Smart-licenses.

NOTE! If you unregister, you will see a Deployment change ready itself. Don't bother deploying this change. CDO will fix this later. 

3. On both devices, change the Management Interface IP to one that is unique to the device, but can gain access to the Internet. The Token process will use this IP for differentiation.

4. Failover to the Secondary device.

5. Add the Secondary device in the CDO portal with Use Registration Key (the other method will not work). Click Next.

6. Give it a unique name. Click Next.

7. Uncheck "Immediately perform security updates....". Click Next.

8. Copy the Registration key.

9. Go back to the FDM. Verify it is Unregistered with Smart Licensing. 

10. Go into Cloud Services and click "Get Started" in the Cisco Defense Orchestrator box.

11. Paste the Registration key and select your region (must match the region on both units!). Click Register.

12. Click Accept (if you accept).

13. Go back to CDO and click Next.

14. If you created a Smart License Token in Cisco.com's Smart License portal (your portal), then, you can paste it here. I recommend doing this, and not skipping. Click Next.

15. Click "Go to devices page" to see the result. Let the device FULLY sync before proceeding.

16. Click the Refresh botton to update, and do NOT attempt any local FDM management or changes while this takes place.

17. Once synced, you should highlight or check the box next to the new device. If the Failover (HA) was configured correctly, CDO should recognize there's a "mate" device. It also recognizes the roles.

18. Failover to the Primary device. Verify it's Unregistered.

19. Highlight or check the previously added Secondary device in CDO. Click "Onboard Device" in the right-hand column. 

20. Give it a unique name. Click Next.

21. Uncheck "Immediately perform security updates....". Click Next.

22. Copy the Registration key.

23. Go back to the FDM. Verify it is Unregistered with Smart Licensing. 

24. Go into Cloud Services and click "Get Started" in the Cisco Defense Orchestrator box.

25. Paste the Registration key and select your region. Click Register.

26. Click Accept (if you accept).

27. Go back to CDO and click Next.

28. If you created a Smart License Token in Cisco.com's Smart License portal (your portal), then, you can paste it here. I recommend doing this, and not skipping. Click Next.

29. Click "Go to devices page" to see the result. Let the device FULLY sync before proceeding.

30. Click the Refresh botton to update, and do NOT attempt any local FDM management or changes while this takes place.

NOTE: You may see your device listed as "Unprovisioned". Give it a minute.

NOTE! CDO will likely fail to recognize the current HA active/standby state. The following procedure addresses this.

TIP: Be sure to watch the View Jobs status on the bottom-right. 

1. Highlight the Primary/Active device in CDO and click High-Availability in the right-hand column. The status will show "Device Busy" for some time. Please be patient.

2. Once the Device Busy goes away, go back to the previous screen. You may now see a Read Error. Click “Read Configuration” on that device.

The main issue is that CDO can't correctly determine the active/standby state of the FTD HA pair. In fact, it seems to indicate the devices are Unreachable, shows Read Errors, and any attempt to Reconnect, Read Configuration, or investigate the active/standby state results in inconsistent results, none of which FDM shows. FDM shows the devices have been stable, but CDO seems to believe the active/standby state is swapping.

[bmurphree@myemma.com]

What we learned was, you simply cannot set the Management interfaces as "Use the Data Interfaces as the Gateway" for connectivity. CDO needs the ability to connect to the device via token, on the Management interfaces. This strikes me as a real problem, not of how this works, but why this seems to be the only way:

1. FTD/FDM refuses any connections to the Standby device.

2. FTD/FDM apparently was not designed to "call-home" to CDO via the external interface like, for example, a Meraki MX100 would. 
3. FDM/FTD does not allow you to set the Management Interface 0/0 to use a unique gateway, and use the firewall as the default gateway without another Layer 3 device (at least that's what I've been told).
4. Even though the tokens are different for the two HA devices, CDO is not smart enough to differentiate the devices and connect to them using the token method.
5. Even after CDO discovers the HA devices and shows them as clustered, in my test, the FTD devices can no longer query the peer.

Not being able to connect to the standby unit from CDO or FDM has causes more problems than it resolves in the architecture and in use (I can't stress that enough). That should be fixed.

[Marvin Rhoads]

I just worked through this in the dCloud lab.

The lab guide had us onboard each FTD separately BEFORE they were in an HA pair. We onboarded using credentials and the management interface address since there was a SDC setup in the environment. Once we had both FTDs onboarded we select the first one and then configure high availability. the steps from there were minimal - setup the failover interface and addressing. After applying that config, the HA creating and syncing happened as expected in about 10-15 minutes.

I have a CDO account also and will try onboarding via token and creating an HA pair there when I get a chance.

I've also asked one of the Cisco TMEs to have a look at this thread and offer any tips.

[bmurphree@myemma.com]

It appears Cisco's documented the procedure for an already active HA pair to be onboarded, and CDO does discover that there is a mate when the first device is onboarded. However, they don't declare that the Management interface, which is always accessible, even if the FTD device is in Standby mode must be used for CDO to remain connected to both devices.

What's debilitating about all this is what Cisco has changed on FTD devices vs. what was supported on the older ASA. Since these new changes, onboarding an HA pair to CDO is far more difficult now, as are a number of other features we might want to use in FTD.

My declaration is that the way the Standby unit cannot be managed outside the M0/0 interface, the fact that you can't manage an FTD device through a Site-to-Site VPN tunnel, the fact that you can't change the SSL portal port from 443 (so you can use VPN Clients to connect), and that the Call-home to CDO can't be sourced from the external interface all make the FTD a truly difficult solution. There seems to have been too much thought put into the API and analytics/threat facilities of FTD and far less into practical everyday use of the FTD.

[bmurphree@myemma.com · Accepted solution]

Solution:

1. Set Management interfaces to use its own routing table (in other words, to use a separate Layer 3 device as its gateway, and should NOT be the FTD device itself, because Cisco doesn't allow you to on the Standby unit)

2. Then follow my initial procedure above.

While this works, it's not reliable. FTD devices may still display an uncertain cosmetic error, "Unable to query (the other device)", while CDO shows no problem.

Needless to say, this solution needs work. While Meraki has a solid technology here, Cisco's is very fragile. Operate with care and careful recovery planning! In other words, risk of failure or misstep is high, so please don't hold me accountable if this procedure fails for you. This isn't fool-proof yet.

[BURKHARD LANDWEHR]

In 2022 it´s also not running via wan interface.......... very bad product, with even badest support.