Hey all,
Another weird one.
TMS - 14.3.2
Monitoring multiple device via HTTPS.
I'm seeing some odd issues from TMS where it seems to think that it has lost HTTPS communication with multiple devices. We have many hundreds of devices that we monitor across multiple organisations in TMS (thankfully, we don't use it to schedule!) such as VCS-Es, VCS-Cs, MXP, C and SX series devices. Yesterday it appeared as though TMS wanted to stop monitor a LOT of these devices. Most of the VCS-Cs, MXP and other devices that seemed to all loose communication are in organisations outside of our main campus.
However, the odd thing is, that they haven't really! Its a "phantom" lost communication!
If you navigate to the connection tab in TMS and hit the "Save/Try" button (in fact you only need to navigate to this page, or the edit setting page, or force a refresh etc.), HTTPS communication is established immediately. In fact, we can also RDP into the TMS box and navigate to any and all devices from a browser with no issues using HTTPS. You can even have a browser session open to a device an have an auto refresh running (say on a camera/call page), with no problems. Further, if you navigate the the affected device (such as VCS Control) and look in the "System --> External Manger" menu, the device shows that our TMS management server is active on HTTPS. However, after a couple of minutes, TMS again thinks it has lost HTTPS connection!!
The really odd thing is that we haven't lost access to all devices. Those on campus are OK, as are the VCS-Es on the core, but a most of the devices in other organisations have got this "phantom" lost communication - weirdly not all of them though!!!
Recently, the backbone network provider installed some DDoS devices on the MPLS network that caused havoc with our neighbour zones, effectively killing our entire VC network. I'm wondering if these devices might also be responsible for these problems? So, I need to know what is different between the TMS HTTPS monitoring communication, and actually forcing a refresh of this connection or navigating to a HTTPS server on a device via a browser.
I have tried a packet capture on TMS, looking at device whose HTTPS response remain unaffected and a VCS Control that is affected, but it show no traffic on either, so I don't really know what I'm looking for (I was expecting a heartbeat or some-such).
Any suggestions, please let me know, before I go back to the core network provider....
Cheers
Chris
