Webex Servicability Node and Expressway Connection

[bspalmer1]

The instructions on getting the two connected are rather straight forward.  I have followed them and have tried both TLS and Non-Tls connections with the same 'timed out' error.  Has anybody gotten this to work?   

I dont see any blocks in our log aggregator at all from the firewalls.  I have seen in a packet capture with TLS that there is certificate exchange and that there is encrypted message passing which I can't read for obvious reasons.

The other method of putting Expressway into the Cloud is via the ECP mechanism built onto it which I am trying to use the Serviceability node for instead.  There are upcoming feature enhancements that would be useful which is why I want to get this working.

TAC hasn't been the most helpful despite providing the PCAP.  This is a newer product and there isn't a lot of details out there around troubleshooting.

[bspalmer1]

I wanted to provide an update.  Cisco TAC after looking at my packet captures setup attempted to replicate my problem.  They did.  There is a bug/flaw in the connection between the webex serviceability connector and expressways.  This would impact secure or unsecure connectivity.  They are looking at this and are going to get back to me on this as they engage in more discovery of the issue.

The goal here is really to connect UC, Expressways, and Cube all together to see if it can give more end to end tracing within the troubleshooting tool in Cloud Connected UC.  This could be a huge improvement in the speed of resolving an issue with a call and while it works with UCM currently having the other pieces integrated could provide a much better overall view.  Stay tuned.

[bspalmer1]

There are two possible fixes currently.  One is to change the administrator port on the Expressway which requires a reboot and will also permanently change the admin URL for it with a port requirement to gain access.  The other which I am waiting on involves IP tables on the expressway.  My guess is some sort of port address translation networking mechanism(More of a hotfix).    The nexus is that you can't connect expressway on the admin port 443 to the troubleshooting node.

It took a case going all the way to development and approx. 5 months or so of back/forth to reach this point.

[Roger Kallberg]

We have ECP for Webex Serviceability and have connections with all of our various Expressways in it without changing any of what you outlined.

[bspalmer1]

We run Expressway 14.0.7 and are on the latest release channel for the ECP.  It was able to verify the server after we had made the port change via Cisco's direction.  Otherwise it has never been able to verify the C or the E in our development environment at any point.

[Roger Kallberg]

That is odd. We did use the same version on Expressway from what I remember when we first defined the systems in ECP. Since then we have upgraded all of them, so at the moment we’re on newer versions.

[bspalmer1]

The issue is with the Architecture of the ECP node.  If using the Explicit Proxy it will send all connections on port 443 to that proxy.  So one fix is to change the admin port on those expressways to 445 for example.  Then all the API requests or admin login urls to Expressway need to take that new port into account.

The other method involves adjusting ip table routing so intranet 443 traffic isn't routed out to proxy and this is done supposedly in the server settings for the expressway on the ECP, but; I am confirming that with TAC/BU/DEV

[Roger Kallberg]

Aha, as we don't use a proxy for our ECP this would not be applicable to our setup. We have it setup to be allowed through the firewall with a specific rule.

[bspalmer1]

To provide another update.  They had a workaround but it required running a script via the CLI which essentially put in ip route tables to handle this and it had to be run every time the connector was updated.  This past week they released an update to the product that would do all this for you it just required a turn off/on to enable this.  All the expressway alerts are gone from the product now and I am testing the analysis feature now.