Webex Teams Hybrid Calls not working to outbound numbers (Urgent)

[PWJPW]

Our hardware VM died and we rehosted our VMs on a new ESXI server.

Everything working great including inbound but I can't get calling PSTN numbers from the Webex Teams app to work.

We exported and imported the guest VMs (CUCM, Exp C and E) so no settings have changed.

When we make a PSTN call from Webex Teams the call doesn't connect and Exp E is showing 404 not found on the final leg. It appears to be trying the Default Zone instead of matching the ciscospark.com search rule so it never reaches my VCS-C server. I think that is wrong?

Call flow: Dialled 0800800150 from my Webex Teams app.

cucmpub.abcd.com is my CUCM server.

james@mycompany.call.ciscospark.com is my Webex Teams URI.

Search history below:

Search (41)
State: Completed
Found: False
Reason: Not found
Type: SIP (INVITE)
SIPVariant: Standards-based
CallSerial Number: 9d09ffea-a894-494b-ad8d-747fe195e6ec
Tag: 699ee272-e8ba-42ef-8bd9-ae80f7831d52
Source (1)
Authenticated: True
Aliases (1)
Alias (1)
Type: Url
Origin: Unknown
Value: james@mycompany.call.ciscospark.com
Zone (1)
Name: Spark_DNS
Type: DNS
Path (1)
Hop (1)
Address: l2sip-cfa-02.wbx2.com:5062
Hop (2)
Address: 127.0.0.1:5070
Destination (1)
Alias (1)
Type: Url
Origin: Unknown
Value: sip:0800800150@cucmpub.abcd.com;user=phone
StartTime: 2019-03-17 11:46:54
Duration: 0.06
SubSearch (1)
Type: Directed
Path (1)
Hop (1)
Address: 62.252.x.x
Hop (2)
Address: cucmpub.abcd.com
SubSearch (1)
Type: Admin Policy
Action: Proxy
ResultAlias (1)
Type: H323Id
Origin: Unknown
Value: sip:0800800150@cucmpub.abcd.com;user=phone;box-call-serial-number=c18a65df-42d0-4990-a97b-05fb59c18ab8
Zone (1)
Name: DefaultZone
Type: Default
Protocol: SIP
Found: False
Reason: Not found
StartTime: 2019-03-17 11:46:54
Duration: 0.06
Gatekeeper (1)
Address: 172.16.31.251:5071
Alias (1)
Type: H323Id
Origin: Unknown
Value: sip:0800800150@cucmpub.abcd.com;user=phone;box-call-serial-number=c18a65df-42d0-4990-a97b-05fb59c18ab8

Is the 404 coming from itself, VCS C or CUCM?

Call Details

Call information
State	Connection Failed
Start time	2019-03-17 11:46:54
End time	2019-03-17 11:46:54
Duration	0 seconds
Tag	699ee272-e8ba-42ef-8bd9-ae80f7831d52
Serial number	9d09ffea-a894-494b-ad8d-747fe195e6ec
Type	Video
SIP variant	Standards-based

Bandwidth
Requested	16064 kbps
Allocated	16064 kbps
Route	Spark_DNS -> Zone005ToTraversalSZ -> TraversalSubZone -> TraversalSZtoDefaultZ -> DefaultZone

Leg 1
Bandwidth node	Spark_DNS
Source alias 1	sip:james@mycompany.call.ciscospark.com (Url)
Target alias 1	sip:0800800150@cucmpub.abcd.com;user=phone (Url)
Protocol	SIP
Address	18.217.166.80:34681
Transport	TLS
Reason	Not found
Cause	404

Leg 2
Bandwidth node	Default zone
Target alias 1	sip:0800800150@abcd.purplewifi.com;user=phone (Url)
Protocol	SIP
Address	62.252.x.x:5062
Transport	TLS

Leg 3
Bandwidth node	Default zone
Target alias 1	sip:0800800150@cucmpub.abcd.com;user=phone (Url)
Protocol	SIP
Address	172.16.31.251:5071
Transport	TLS
Encryption type	None

Session 1
Status	Replaced
Media routed	False
Call routed	True
Participant 1	Leg 1
Participant 2	Leg 2
Bandwidth allocated	0 kbps
Bandwidth requested	0 kbps

Session 2
Status	Failed
Media routed	False
Call routed	True
Participant 1	Leg 1
Participant 2	Leg 3
Bandwidth allocated	16064 kbps
Bandwidth requested	16064 kbps
Route	Spark_DNS -> Zone005ToTraversalSZ -> TraversalSubZone -> TraversalSZtoDefaultZ -> DefaultZone

Zones

DefaultZone Default zone 0 0 kbps Off On 
B2B External DNS 0 0 kbps Off On Enabled search rules: 1 
Spark_DNS DNS 0 0 kbps Off On Enabled search rules: 1 
Spark_to_VCS-C Traversal server 0 0 kbps Off Active Enabled search rules: 1 
Unified Traversal Unified Communications traversal 0 0 kbps Off Active Enabled search rules: 2; disabled search rules: 1
VCS-C Traversal Zone Traversal server 0 0 kbps Off Active No search rules configured 
Webex Zone Webex 0 0 kbps Off On Enabled search rules: 1 

Search Rules

49 Webex DNS SIP Any No Alias pattern match Regex (.*)@(.*)(\.webex\.com).* Replace Stop Webex Zone All SIP Variants 
50 Incoming SIP Any No Alias pattern match Regex ^\S+@abcd.com Leave Stop Unified Traversal All SIP Variants
51 Incoming2 SIP Any No Alias pattern match Regex ^\S+@cucmpub.abcd.com Leave Stop Unified Traversal All SIP Variants
52 B2B External Any VCS-C Traversal Zone No Alias pattern match Regex ^\S+@\S+$ Leave Continue B2B External Any
100 From Spark Hybrid to Exp-C SIP Spark_DNS No Any alias Stop Spark_to_VCS-C All SIP Variants
100 From Unified CM to Spark Hybrid SIP Spark_to_VCS-C No Alias pattern match Regex .*@.*\.ciscospark\.com Leave Stop Spark_DNS All SIP Variants
102 LocalZoneMatch Any Any No Any alias Continue Unified Traversal Any

I really appreciate your help.

Thanks

James

[PWJPW]

Really sorry to bump but I have until the morning before the office re-opens after the weekend and really looking for some community help.
Thanks
James

[DON PETERSON]

We're having the exact same problem.  But we had it working, upgraded the Expressway and CUCM servers to the latest version then it stopped working.  No matter what I do, inbound calls from Webex and Webex Teams hit the default zone (search rule) instead of the hybrid inbound rule.  I have a case open with TAC but it's going on 4 days now and they have nothing.  I'll update this if we make any progress.

[mlong0000]

Hi, did you find a cause for this problem? I'm experiencing the same.

[PWJPW]

We did solve it, in our case it was due to the network setup as it was natting and thus sending the traffic to the wrong place. I fixed it my enabling the IPv4 static NAT mode on the System > Network > IP page. Set both of these:

IPv4 static NAT mode: On

IPv4 static NAT address: Your public IP

[mlong0000]

Thanks for the reply James. Unfortunately I already have that setting configured.

[Nathan Grovhoug]

Any update on this? I am experiencing the same problem setting up hybrid calling for WebEx devices. All calls from Webex to the Expressway E are hitting the default zone instead of the WebEx Zone. Expressway 12.5.9

[Nathan Grovhoug]

OK. I found out the problem with this for me. The Public SRV record for _sip._tcp.<domain> was pointing to 5061. once I updated the entry to 5062 the webex zone started catching the inbound. Hope this helps someone else.

[Ayodeji Okanlawon]

Hi Nathan,

You should not be deploying WebEx hybrid calls today. You should be using one of the offerings with WebEx Teams...

1. Webex Teams ( CUCM calling mode)

2. WebEx Teams (Jabber Mode)

Both of these options will allow team client to register as a CSF device in cucm and will use your normal MRA discovery setup to locate your expressway.

[Nathan Grovhoug]

Thanks for the reply. We are using teams with cucm calling mode but we needed a way to allow WebEx room devices in the cloud (e.g. room kits) to dial out using cucm services. Is there an alternative solution that I am missing?

[Joshua O'Connor]

For devices hybrid calling is still the process required to allow endpoints to be integrated. Only other option would be registering to CUCM still and then using edge for devices to make them cloud aware. Really depends on your needs. Also this doesn’t provide a personal device experience to cloud registered. Only really useful with Places