CWMS 2.6.1.39 MeetingAdmin hangs when attempting to open page

[Clifford McGlamry]

Installed CWMS v2.6.1.39 for Customer POC.  The MeetingAdmin page works normally when the server is started, but then gets into a hung state.  Attempts to connect to it leave the browser just spinning while attempting to connect.  Oddly enough, the Meeting pages for the users still work normally at this juncture.  

I cannot find any bugs listed or discussions about this.  Restarting the server fixes the problem temporarily, but I'm unclear how to troubleshoot this.  No guidance that I have located in any documentation seems helpful thus far.

Any suggestions appreciated.

TIA,

Cliff

[dpetrovi]

Hi Cliff,

Can you share more details about your deployment, please?

What is the size of your CWMS deployment? (50, 250, 800 or 2000 users)

Do you have Internet Reverse Proxy (IRP - Public Access)?

Do you have High Availability (HA)?

When you ping Admin URL, to what IP address does it resolve?

When the issue is happening, are you able to ping that address?

When the issue is happening, are you able to run : telnet adminURL 443    from your client machine to see if you can establish connection on port 443?

Is your WebEx Site URL accessed using the same IP?

Thank you.

-Dejan

[Clifford McGlamry]

POC is a 50 user deployment that does not have HA.  There is an IRP deployed. 

When you connect to the address, it does detect that it needs to go to HTTPS, and shifts, and that's where it hangs.  There is a separate IP address set up for MeetingAdmin than the CWMS server itself.

The Webex site URL points to the IRP. 

Haven't tried the telnet trick, but since it's shifting over, I suspect it's working, but I'll have to try tomorrow.

[dpetrovi]

Hi Clifford,

Thank you for this information. Since you have a single Admin VM, your Admin URL is hosted on the Private VIP address  on this Admin VM. Since WebEx Site URL points to Public VIP address on IRP VM and access is working indicates internal modules on Admin VM are fine, but something with the Admin URL is not loading.

When you said restart of Admin VM resolves the issue, does that mean that the issue returns soon after? 

Also, on your PC, try to edit the hosts file to add WebEx Site URL to point to Private VIP, so when the issue with Admin URL access happens, you test access to WebEx Site URL but this time hosted on the Private VIP (just to see if it will be accessible or not).

Is this POC in some sor tof a DMZ, or it is internal to the network?

Have you tried different browsers?

-Dejan

[Clifford McGlamry]

It's still happening.  We've determined that when this occurs, there is zero connectivity to the MeetingAdmin URL.  If you attempt to telnet to it on the https port, it just fails.  Wireshark shows no packets coming back.

Rebooting fixes it temporarily, but it freezes again after a period of time.

This POC is all on the internal network.  

[dpetrovi]

Hi Clifford, 

Is the Private VIP address pingable during the outage?

The only time I've seen something like this is when a switch/router on the network hijacks the IP address and doesn't update the ARP table with the MAC of the Eth1 on Admin VM, but instead it is using it's local MAC address. 

The only way to investigate this further is for you to create Remote Support Account (CWMS Administration > Support > Remote Support Account > username 'ciscotac', validity 30 days) and save the Passphrase somewhere on your PC. Once the issue happens (hopefully within 30 days), open a TAC ticket, provide the passphrase so that TAC engineer can create remote Support Account password, SSH to Admin VM and see what is happening. 

If the issue is coming back regularly, you can also collect Overall System Logs from the system for the time of the outage (of course , after your restart the VM).

Unfortunately, if Private VIP is not accessible via  telnet on 443 during the outage (connection refused) but is pingable, without SSHing to the system on Eth0, you don't have any options for troubleshooting and restarting the VM is the only option.

-Dejan

[Clifford McGlamry]

It's pingable....just can't connect. 

Okay.  This gives me some things to look at.  This one is harder because their infrastructure is so locked down (financial house). 

[Chenzhe Zhang]

Was this issue resolved in the end? I am having exactly the same issue.

Regards

Chenzhe

[Clifford McGlamry]

Turned out to be a dns  issue (dns didn't match the up address of the servers).

[Chenzhe Zhang]

Thanks Clifford. I was wondering about DNS entries as well. Can you maybe elaborate a bit more about what the mismatch was please? 

For the WebexAdmin page, it is just a private VIP address behind the admin URL right? We have this entry in the internal DNS forward zone. Do we need anything else to fix this issue?

Regards

Chenzhe