We have two Firepower 4145 firewall appliances that are clustered together on our network, each node is located in two separate data centers (data center A and data center B). Below the cluster (inside interfaces) we have a pair of Catalyst 6880 switches virtualized (vss) and above the cluster (outside interfaces) we have two Catalyst 9500 stacked switches (stack with virtual stack links). The cluster control links also connect to the pair of stacked Catalyst 9500 switches. On a normal day of operations this design works great and we can upgrade the software on the clusters, forcing failover from the control node to the standby node and then back again with out a problems. I have attached a diagram of the devices in question.
With in the past 6 months we have encountered complete power failures in both of our data centers. The first power outage was in Data Center B. When this happened, we anticipated the control node would continue to function and pass traffic. However, the control node (node 1) went into a failure state and stop forwarding traffic. The switches above and below the cluster were functioning as expected, but could not get the firewall cluster node 1 to work. Once power was restored to Data Center B, node 1 continued to be in the same state of not forwarding traffic and node 2 was not forwarding traffic either. We cycled the power on node 1 and when it finished booting both nodes joined the cluster and starting functioning properly.
On the second power outage just a few weeks ago, power was lost in Data Center A and the same situation occurred. Node 2 stopped forwarding traffic, but the switches above and below node 2 were functioning. Once power was restored to Data Center A, we had to go back to Data Center B and power cycle node 2 in order to get the firewalls to start passing traffic once again.
Unfortunately we did not have time to troubleshoot this problem or capture and logs while the event was happening. When the node in the data center lost power, the logs were lost, and when we had to power cycle the non-functioning node, the logs were also lost. FMC only showed the cluster was down and no members in the cluster. After the power cycle and completion of the failed node re-booted, FMC just saw the nodes come up and joined the cluster. So little to no forensic information is available.
Since we can force failovers gracefully during software upgrades without problems, I don't think it's the failover routine. My suspicion is when one of the Catalyst 9500 loses power, it is has some sort of problem on the cluster control links and causing the node that is supposed to become the single node member of the cluster doesn't happen. That is the only difference I can see between doing a graceful failover during software upgrades and a complete power failure of the devices above and below the firewall cluster. When power is restored to both nodes, a power cycle has to be performed to stimulate the node in the bad (non-forwarding) state in order for it to come up and start functioning again.
I hope I was able to articulate this problem into and understandable narrative. If anyone has any ideas, I welcome them. If someone is knowledgeable with the exchange between the cluster nodes on the cluster control links, I would appreciate your input or your speculation of why our firewall cluster does not failover properly during a power outage.
