Jabber Expressway MRA connects to all but Softphone

[kdotten36]

Looking for some assistance.  I have Expressway 8.7 in the recommended two-legged design except not using NAT since the outside interface is already publicly accessible.  I have a valid Cert on exp-e, and a privately created CA cert on exp-c using the Certificate Guide and OpenSSL.

Getting Failed to Establish SSL errors.  In the debugging logs I see "peer did not return a certificate".  I have my private CA and self-signed CM cert installed to the Enterprise Trust store on the laptop and the iPhone from which I'm connecting.

CUCM (9.1.2) has a tomcat cert also signed by the private CA, and the CallManager cert is self-signed.  The root private CA and self-signed CM certs are uploaded as Trusted CA on both expressways.  I've tested Jabber for iPhone and Jabber for Windows and both can successfully register externally with Exp-E, IMP, CUX, but not the softphone.  I have a SIP Trunk for outbound calls and the Security SIP Profile has a separate port configured and there are two traversal zones, one for UC and one for that purpose.  I also tried removing the Trunk's traversal zone and all references to it in case it was interfering and got the same results.

I'm attaching one of the Jabber Problem Reports and a Developer's log from expressway E.  The time of the registration failure is 16:30 on 2/4/16.

If more are needed I can run another.  Thanks for looking.

EDIT: Forgot to mention, I also tried adding a firewall rule to allow all traffic to and from the expressway temporarily and it still didn't work, so it shouldn't be a firewall oversight.

[Chris Deren]

Are the CSF and TCT devices configured and associated with the end user? Do they work on internal network? Registration of soft phone has nothing to do with SIP trunks, etc. those are only used for B2B video across Expressway. 

[kdotten36]

Yes, the devices were configured correctly and work on the internal network.  I understand the SIP Trunks are not used for MRA, I only mentioned it because it's a common mistake have the SIP ports conflict between neighbors so I was trying to convey all available detail.

As coincidence would have it, I was able to finally get it working about an hour ago.  The problem was the Default Zone configuration on the C and E.  I configured both Expressways strictly by the guides (Basic Configuration, Mobile and Remote Access via Expressway, and Certificate Creation and Use Deployment Guide) from factory defaults, and both Default Zones were set with an Authentication policy of "Do not check credentials" and Use Default Zone access rules on port to "TLS (5061) and MTLS (5062)".  After picking apart the Admin Guide to better understand these settings, I changed those to the following...

Expressway-C - Default zone

Authentication policy = Check credentials
SIP Media encryption mode = Best effort
Use Default Zone access rules on port = MTLS Only (5062)

Expressway-E - Default zone

Authentication policy = Treat as authenticated
SIP Media encryption mode = Best effort
Use Default Zone access rules on port = MTLS Only (5062)

That fixed it.  Devices now register and calls are flowing.  What strikes me as odd is that none of the documentation mentions this, unless I'm mistaken, I even specifically searched for recommended Default Zone settings for MRA when I began to suspect it.  I guess it's possible some non-standard network config on our end made my setup an exception, but I would think it's an important step to cover if it is indeed required.

Thanks for the response though.  I'm a big fan of CDW (you're our Cisco partner!), and sorry to be so wordy, just trying to help out the next dud who failed miserably at this for as long as I did.

[Chris Deren]

Good job, and thanks for the update. I find it interesting that the zone was ACTIVE and you were able to connect for IMP/UCXN but not CUCM, I would think all services would not be available.  

[kdotten36]

I don't think they use SIP to Expressway, just CUCM.

[Ayodeji Okanlawon]

Hi,

I know this has been resolved but I wanted to chip in a little here..

I have done a few MRA solutions and I have never touched the default zone. None of your MRA call traverses the default zone. I am not sure the relevance of this zone in the whole equation.

Authentication policy affects endpoints registering to a zone. As we know Expressway solution doesn't register endpoints. Not sure what is going on, but logically these should  not impact your setup. If you have time, you can revert to the old configuration and check again..

[kdotten36]

Gladly, I'd like to know it's been legitimately resolved as well.  I will undo my changes and report back in a bit.  The only other change I made prior to adjusting the default zones was deleting expired certificates from CUCM (not regenerating, that had already been done well prior to the successful tests), but as I'm not doing TLS from CUCM to exp c, that was just a shot in the dark.  I ran one test that night after doing so and it still failed anyway.

Will update soon.

[kdotten36]

I reset the DefaultZone to default settings and it reverted to the behavior of not being able to login to Phone Services, but IM&P and CUX still worked.  I edited one setting at a time and narrowed it down to this one...

DefaultZone > SIP > Use Default Zone access rules on port = TLS (5061) and MTLS (5062)

When this is set on the Expressways, I cannot login to phone services.  When it is set to "MTLS Only", I can.  The other settings (authentication policy and Media encryption mode) can stay at default.  I have no Default Zone access rules configured on either expressway.

The error I see on exp-e when logging on with the TLS and MTLS setting is below

tvcs: Event="Inbound TLS Negotiation Error" Service="SIP" Src-ip="jabberclient-ip" Src-port="2766" Dst-ip="expe-external-ip" Dst-port="5061" Detail="peer did not return a certificate" Protocol="TLS" Level="1" UTCTime="2016-02-08 15:53:11,739"

This error is repeated multiple times for each client.  I tested with two, a J4W client and an android.  Let me know if there is anything else you'd like to see.

[Ayodeji Okanlawon]

Okay that is interesting to know and thank you for taking the effort to do this. My default configuration already has MTLS and I am running X8.6.

I will keep an eye on this setting and will try and verify if we can only use MTLS

[kdotten36]

I see.  I'm running X8.7 and the informational pop-up does show that MTLS Only should be the default setting.  It's entirely plausible that I changed it at some point during my troubleshooting and never set it back after correcting other errors, since I did initially have a certificate problem on my CUCM.  In that case, it's my oversight.  Appreciate you chiming in.

[Ayodeji Okanlawon]

Oh great, I think the picture is clear now. This is a layer 8 error :)

Nice one. I love it when we try to get to the bottom of things, that's what separates us from the crowd.

[kdotten36]

Quick question since you seem to be the primary MRA responder around here...

My last issue is with Jabber for iPhone.  For months I thought we had a routing issue that prevented it from registering 50% of the time because we have a subscriber on a vlan in which the device IP itself cannot communicate.

I recently discovered that it was because of my method of testing, not the system's inability to fail back to the publisher that was my issue.  I would successfully register to CUCM, then force terminate the running app on iPhone, then relaunch and it would fail.  If I'm correct, this activates a keepalive timer that makes the CM reserve my registration, essentially rejecting my next attempt.  This also affects the transition from internal network to MRA, as it will fail to register when the network switches.

Odd thing is, Jabber for Android/Windows don't have the same problem.  They can toggle back and forth and reregister even though they're using the same SIP Profile as iPhone with all the Timers set at the guide's recommendation, which are as follows...

• Timer Register Delta to 120
• Timer Register Expires to 720
• Timer Keep Alive Expires to 720
• Timer Subscribe Expires to 21600
• Timer Subscribe Delta to 15

Any idea why iPhone would fail to re-register while other devices can?  Or, is there a more ideal set of timers to be applied to each device than what is found in the Jabber Deployment Guide?

Thx.

[Ayodeji Okanlawon]

The honest truth is I do not know. What I have observed is that sometimes when I disconnect from LAN to use wireless (all on MRA) sometimes my jabber for windows doesn't softphone doesn't re-register. What I have observed is that there are several sessions opened on the expressway-E and it appears that this has an impact on registration. I have not had time to look into it in detail

[kdotten36]

Interesting.  I think while registered via MRA those settings might be related to the Advanced section of Unified Comm page on Expressway-C, which defines how many times a client can authenticate during a given interval and how often it cleans up expired sessions.  I had to adjust these settings during initial testing since I was reconnecting an unusual number of times.

For me, it's happening whether I'm switching from internal LAN to cellular, if the app is forced closed, or if I lose service (like getting on an elevator).

What's strange is that it only happens on iPhones.  I have to terminate the app and wait 12 minutes before CUCM will let it register.  I've also tested J4W, Android and even iPad, and each is able to eventually recover its registration albeit sometimes with a delay, which is acceptable.

I opened a TAC case on it and will see where it leads.  I've since discovered that after failing registration, if I reset the TCT device on CUCM, it will then succeed.  The device is also usually (not always) still showing as Registered on CUCM even though it's not.

[kdotten36 · Accepted solution]

Wish I hadn't wasted so much of my own time on this, but it is now resolved.

Device Config > Built In Bridge needs to be Off.  It was set to Default which is On by a Service Parameter on our cluster.

The engineer said it's the second case he'd seen of this nature and will submit for it to be included in future deployment guides.

Registrations are now clean after network changes, service outages and app terminations.