ISE distributed deployment nodes freeze but no clue about what has happened

[ajtm]

Four nodes distributed in two Datacenters (all Virtual Machines).

Datacenter1: Node 1:Pri Adm+Pri Mnt; Node 2:PSN1; Datacenter2: Node 3:Sec Adm+Sec Mnt; Node 4:PSN2

Last week both nodes in Datacenter1 become unresponsive - no WebUI access, SSH access working but unable to login.

All RADIUS requests, in both PSNs, receive authz failed reply (AD and internal users)

TACACS+ failing in PSN1 but working fine in PSN2 (internal users only)

Only way we were able to restore service was to reload Datacenter1 nodes (using VMware).

Not able to find any evidence of what has happened in the logs bundle - no logs at all during the incident.

Not understanding why PSN2 was affected.

[Surendra · Accepted solution]

Seems like a classic VMWare Snapshots or Hot vMotion playing a role. I would suggest to check with your VM team if any of the mentioned operations were done on those VMs on that day.

[Marvin Rhoads · Accepted solution]

This symptom can occur when a snapshot of the VMs occur due a storage backup.

ISE VMs cannot be backed up with snapshots while they are running.