I need advice from experts on this forum regarding a very strange issue that occurred in March 2025 at my current workplace, one month before I joined the company.
We have a Cisco ISE 3.1 Patch 10 cluster deployed across two data centers as follows:
-
ise01: Primary Admin / Secondary MnT (Data Center A)
-
ise02: Secondary Admin / Primary MnT (Data Center B)
-
ise03: PSN (Data Center A)
-
ise04: PSN (Data Center A)
-
ise05: PSN (Data Center B)
In March 2025, we performed a Disaster Recovery (DR) exercise that required completely cutting off connectivity to Data Center A. Prior to disconnecting Data Center A, we promoted ise02 to Primary Admin / Primary MnT, resulting in the following state:
-
ise01: Secondary Admin / Secondary MnT (Data Center A)
-
ise02: Primary Admin / Primary MnT (Data Center B)
-
ise03: PSN (Data Center A)
-
ise04: PSN (Data Center A)
-
ise05: PSN (Data Center B)
After the promotion, everything looked good, and we cut off connectivity to and from Data Center A. During the outage, we were able to successfully authenticate 802.1X on both wired and wireless networks between ISE and Cisco switches and Cisco wireless controllers. MAC Address Bypass (MAB) also worked as expected.
After completing the DR exercise, we restored network connectivity to and from Data Center A. Once connectivity was restored, the ISE cluster appeared healthy, with all nodes in green status. The following day, we promoted ise01 back to Primary Admin.
At that point, everything appeared normal; however, we discovered that all MAC addresses in the MAB database were missing, which caused a widespread outage affecting printers and Cisco IP phones. To recover, I restored a previous ISE backup to a lab evaluation instance, extracted the MAB database, and restored it to the production environment.
We opened a support case with Cisco, but the root cause was inconclusive. Since then, we have upgraded the ISE environment to ISE 3.3 Patch 7.
We are planning another DR exercise in approximately five weeks, and I am very nervous about failover and failback with Cisco ISE. What is the likelihood that this issue could occur again?
Thoughts?
