We have been running SAML SSO for nearly three years no with no issues.
This week we lost Single Sign On for a day. Got this error:
Invalid SAML response. This may be caused when time is out of sync between the Cisco Unified Communications Manager and IDP servers. Please verify the NTP configuration on both servers. Run "utils ntp status" from the CLI to check this status on Cisco Unified Communications Manager
Checked our NTP, we were rocking at Stratum2, within 30ms.
Reached out to our IdP, they said they are authenticating to all other (Microsoft) sites just fine, But one other site that also is listed as CUCM has called in with the same issue.
I reached out to them, and found that at the same time, we both received the same type of errors in our RTMT Alerts:
MatchedEvent : Nov 16 23:32:27 Server1 user 2 platform: Setting the hardware clock FAILED. This failure may be due to a dead CMOS battery or a problem in the hardware clock counter and should be addressed ASAP.
AppID : Cisco Syslog Agent
MatchedEvent : Nov 16 23:33:10 Publisher user 2 platform: The local NTP client is off by more than the acceptable threshold of 3 seconds from its remote NTP system peer. The normal remedy is for NTP Watch Dog to automatically restart NTP. However, an unusual number of automatic NTP restarts have already occurred on this node. No additional automatic NTP restarts will be done until NTP time synchronization stabilizes. This is likely due to an excessive number of VMware Virtual Machine migrations or Storage VMotions. Please consult your VMware Infrastructure Support Team.
AppID : Cisco Syslog Agent
-We both received these errors within 10 minutes of each other.
- We both use internal NTP servers at Stratum1
-We are States apart
-We both went down AND came back up within 10 minutes of each other
-The only commonality is we both use the same IdP.
-Identity Provider states nothing was changed on their side, I tend to believe them, they have nothing to do with our internal NTP settings.
-We both came in the next morning to tackle the issue, and we were up and running with zero SSO issues.
Have you ever seen anything like this? It’s like China turned off the NTP satellites for a day to mess around
