Re: Direct Routing for Microsoft Phone System with Cisco Unified Borde - Page 2

takahashi-ta · ‎02-23-2021

I am using vCUBE (on CSR1000v) to build direct routing for Microsoft Teams Phone System.
ITSP is using Twilio.
In this environment, I can make and receive outgoing calls from Microsoft Teams Client to Twlio direction, but the signaling is not working well from Twilio to Microsoft Teams Client direction.
I used TranslatorX to draw a SIP ladder sequence from the results of the debug ccsip messages and found that the CUBE is not receiving any response to the INVITE it sent to the Micorosoft Teams Phone System.

The TLS connection is normally working as shown below.

(ex)
Remote-Agent:52.114.76.76, Connections-Count:3
Remote-Port Conn-Id Conn-State WriteQ-Size Local-Address TLS-Version Cipher Curve
=========== ======= =========== =========== ============= =========== ============================== =====
5061 140 Established 0 x.x.x.x TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 P-384
3521 142 Established 0 x.x.x.x TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 P-256
3520 141 Established 0 x.x.x.x TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 P-256

However, the moment I make a call from Twilio to the Micorosoft Teams Client, I get the following message.

%SIP-3-INTERNAL: Connection failed for addr=52.114.76.76, port=5061, connId=140

For some reason, the TCP TLS connection seems to be failing momentarily, and I suspect this is why the signaling is not working properly.

Is there any configuration I should review?

By the way, I am referring to the following document for setting up the CUBE.
https://www.cisco.com/c/dam/en/us/solutions/collateral/enterprise/interoperability-portal/direct-routing-with-cube.pdf

rchaseling · ‎04-06-2021

For me.....

1. Yes - static routes configured. defaut route pointing to internet - only static routes are for CUCM and NTP IPs

2. SIP AGL turned on (1 to 1 NAT configured on firewall)

3. Yes - if I didn't nothing would work as I have a firewall north and south of CUBE

4. i disabled revocation completey on those certs as I dont currenly have outbound HTTP allowed on firewall. If I enable revocation is brings down the trunk as you saw

Cheers

Francesco La Bella · ‎01-20-2022

Hi, I know this is a very old post, but I'm facing the same issue (TLS connection from CUBE to MS Teams flapping), and it would be very helpful to know is you have solved the issue in some way.

Thanks in advance,

Francesco

rchaseling · ‎01-21-2022

HI,

Still seems to happen but have not had any issues logged so might just be a microsoft thing?

Carl Ratcliffe · ‎01-21-2022

Hi,

I also constantly get these errors, but the same not had any user facing issues, at least they haven't reported any. Its the same for 8 global Direct Routing CUBE SBCs that i have deployed.

The errors are always 52.114.x.x which I think is the signalling IP range and the media is usually 52.112.x.x or 52.113.x.x.

Is it really an issue ? I would say it is as you don't get connection failed messages for nothing.

Will it cause impact ? possibly not just because there is so much resiliency built in. Not sure if you have ever looked at Microsoft direct routing call reports but you could never tell from these if a call has a problem or not because you get many success=no calls but they are not that straight forward as a success=no can literally be someone just hanging up.

What can yo do about it ? - not much because Cisco will just say its the connection to Microsoft speak with them and Microsoft will say speak to your SBC vendor !

Thanks, Carl

Francesco La Bella · ‎01-28-2022

Thank you all for your answers/opinions.

You are right, I also have no user facing issues but this is just because we have 3 different dialpeers towards Microsoft, so it's really unlikely that all tree dialpeers are busied out at the same time but, it's not impossible... So I think that the behavior is not normal and from packet captures, TLS sessions are always closed by MS side. I opened a case to Cisco TAC and to Microsoft Support but both vendors have no idea about the root cause of the issue....

If I get some news I'll keep you informed.

Francesco

Scott Leport · ‎01-21-2022

Hi,

I concur with @Carl Ratcliffe and @rchaseling i get these error messages all the time too and I’ve done about 3 more of these deployments since my first post. It is not related to specific models and not likely to be IOS-XE version either as I’ve seen them on Cat800V and ISR4K’s running 17.3.X and above.

Not had any reports or complaints from these users either.

i.pletenev · ‎04-27-2022

Hi,

we also have same issues. but in our case we do have some end user problems. i tested the following scenario:

MS user A calls an external user B (MS establish a TLS Session to the CUBE) .

User B answers and the call is running for about 3 or 4 minutes.

After 3-4 minutes MS closes the established TLS Session, but the call is still running (which is ok, because at this moment there are only UDP streams between users)

User B press the Hold button (this forces CUBE to send a SIP message)

CUBE can't send the Re-Invite to the MS because the TLS Connection is already closed

The call is disconnected.

could you guys test the same scenario?

Carl Ratcliffe · ‎04-28-2022

Hi, you can try the following bug fix which worked for me for very similar issues.

CSCwa81136 : Bug Search Tool (cisco.com)

Thanks, Carl

b.winter · ‎04-28-2022

I can confirm this.

A colleague of me said, this did the trick for him, after troubleshooting with TAC.

robetrem · ‎09-01-2022

I would like to know if the workaround as specified by CSCwa81136 (Under tenant configuration section associated with the remote UA, add "no conn-reuse") have worked for everyone's that encountered the %SIP-5-TLS_CONNECTION: Connection failed on sip#.pstnhub.microsoft.com. I have configured the <no conn-reuse> without any positive changes ! still have a lot of Connection failed

derek.fraser · ‎12-22-2022

I'm experiencing the same treatment with constant %SIP-5-TLS_CONNECTION failures running a 4431 and IOS-XE 17.7.1a. Below is a short sampling but it's occurring constantly.

Dec 22 10:03:08.474: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.132.46, port=36416, connId=16067
Dec 22 10:15:11.409: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.75.24, port=60737, connId=16118
Dec 22 10:18:11.408: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.75.24, port=60736, connId=16121
Dec 22 10:22:08.338: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=5061, connId=16081
Dec 22 10:22:38.335: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=15296, connId=16122
Dec 22 10:22:54.455: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.76.76, port=5061, connId=16127
Dec 22 10:30:41.689: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=6464, connId=16128
Dec 22 10:37:41.731: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=6465, connId=16129
Dec 22 10:37:41.731: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=5061, connId=16127
Dec 22 10:38:24.454: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.76.76, port=5061, connId=16133
Dec 22 10:39:59.732: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.132.46, port=17024, connId=16132
Dec 22 10:43:08.760: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=55168, connId=16134
Dec 22 10:46:08.771: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=55232, connId=16135
Dec 22 10:50:41.383: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.75.24, port=5061, connId=16110
Dec 22 10:50:41.384: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.75.24, port=60738, connId=16124
Dec 22 10:50:54.577: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=16140
Dec 22 10:51:41.380: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.75.24, port=60352, connId=16137
Dec 22 10:53:38.766: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=5061, connId=16133
Dec 22 10:53:54.467: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.76.76, port=5061, connId=16143
Dec 22 10:54:38.775: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=55233, connId=16136
Dec 22 10:58:08.125: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.32.169, port=5061, connId=13417
Dec 22 10:58:38.123: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.32.169, port=39104, connId=16119
Dec 22 10:58:43.021: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.75.24, port=8704, connId=16142
Dec 22 10:59:08.107: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.32.169, port=38465, connId=16026
Dec 22 10:59:35.056: %SIP-5-TLS_CONNECTION: Connection failed for addr=52.114.76.76, port=63744, connId=16144

Taking captures we see the OPTIONS ACK from Microsoft (sometimes) take over 1sec to return. When this occurs the CUBE will busy out the dial-peer. The default timer believe is 500ms but would rather not just increase to mask the problem. I'm using DNS as noted previously in this thread to sip.pstnhub.microsoft.com, sip1, sip2. As others have mentioned with the 3 dial-peers for redundancy most of the time end users don't notice and calls work fine. But if all three experience issues an outbound call from Teams>CUBE won't complete. As it's just POC I might try upgrading to the latest code, I had another case where I implemented the 'no conn-reuse' for a 15 minute refresh timer issue between Teams>CUBE>CUCM. @robetrem Both before and after the no conn-reuse these errors persisted so I'm seeing the same as you. Didn't know if others had found a resolution or info to the %SIP-5-TLS_CONNECTION issue they were willing to share.

rchaseling · ‎12-22-2022

Still happens to all the deployments I have. Four out of four have the issue. Happens on 4400s and 8000V - with NATing and without.

derek.fraser · ‎12-22-2022

Thanks @rchaseling for the input, do you end up seeing your dial-peers busy out and how do you end up tackling? After reading that bugID from @Carl Ratcliffe I saw it was to have no conn-reuse under the tenant, I had it globally in voice service voip so I tried that but no change. Not sure what the solution is roll the dice that not all 3 Microsoft dial-peers will error at the same time. Or increase keepalive timer (trace I was seeing is the ACK would come back in like 1.2sec), remove the keepalive altogether from the dial-peer?

Looks like the same treatment on IOS-XE 17.9.2a but now changed to SOCKET_REMOTE_CLOSURE, here are the same but different logs after upgrading.

Dec 22 15:00:55.352: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.76.76, port=45696, connId=92
Dec 22 15:03:28.216: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=68
Dec 22 15:04:28.194: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=25216, connId=95
Dec 22 15:04:41.178: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=98
Dec 22 15:07:57.028: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=3312, connId=99
Dec 22 15:07:57.035: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=98
Dec 22 15:08:11.211: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=101
Dec 22 15:08:57.015: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=3313, connId=100
Dec 22 15:09:25.359: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.76.76, port=46017, connId=97
Dec 22 15:10:57.252: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=101
Dec 22 15:11:11.224: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=105
Dec 22 15:11:27.237: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=19072, connId=102
Dec 22 15:11:57.236: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=19073, connId=103
Dec 22 15:13:27.234: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=105
Dec 22 15:13:41.203: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=107
Dec 22 15:14:27.024: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=1344, connId=106
Dec 22 15:15:55.373: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.76.76, port=46016, connId=96
Dec 22 15:22:29.442: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.132.46, port=64768, connId=39
Dec 22 15:28:55.374: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.76.76, port=5061, connId=90
Dec 22 15:29:11.100: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.76.76, port=5061, connId=118
Dec 22 15:31:28.734: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.132.46, port=36416, connId=117
Dec 22 15:33:27.234: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=107
Dec 22 15:33:41.201: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=121
Dec 22 15:34:27.233: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=17024, connId=116
Dec 22 15:36:00.734: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.76.76, port=35392, connId=120
Dec 22 15:38:52.914: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=121
Dec 22 15:39:00.753: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.76.76, port=35520, connId=119
Dec 22 15:39:22.913: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=6273, connId=123
Dec 22 15:39:41.245: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=126
Dec 22 15:39:52.914: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=6272, connId=122
Dec 22 15:41:57.750: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=126
Dec 22 15:42:11.217: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=128
Dec 22 15:42:57.743: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=55616, connId=127
Dec 22 15:42:59.451: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.132.46, port=62912, connId=113
Dec 22 15:44:00.772: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.76.76, port=35456, connId=124
Dec 22 15:45:31.217: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=60800, connId=129
Dec 22 15:45:51.225: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=128
Dec 22 15:46:51.235: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=60801, connId=130
Dec 22 15:50:41.216: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=133
Dec 22 15:54:01.411: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=42944, connId=134
Dec 22 15:55:51.419: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=42945, connId=136
Dec 22 15:56:51.407: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.75.24, port=5061, connId=133
Dec 22 15:57:11.226: %SIP-5-TLS_CONNECTION: Connection successful for addr=52.114.75.24, port=5061, connId=139
Dec 22 16:02:50.842: %SIP-5-TLS_CONNECTION: Connection closed due to SOCKET_REMOTE_CLOSURE for addr=52.114.76.76, port=5061, connId=118

Greg Brunn · ‎04-07-2023

Hello,

We get this escalation alot so I figured I would head this off in the support forums.
What your seeing happening here is the azure side closing a stale connection after 2 minutes.
I have never seen this impact calls but just shows up in the logging.

Here is the documentation that calls this behavior out.

https://learn.microsoft.com/en-us/microsoftteams/direct-routing-protocols#timeout-policies.

"SIP proxy TCP idle timeout is two minutes. The timer resets when a SIP transaction occurs over the connection."

"After two minutes of idling, (FIN, ACK) is transmitted to the supplier SBC by SIP Proxy within approximately 10 to 20 seconds."

On the Cisco side of the house you can see these remote closures increment up using the following command.

sbccube1#show sip-ua connections tcp tls brief
Total active connections : 11
No. of send failures : 0
No. of remote closures : 0
No. of conn. failures : 0
No. of inactive conn. ageouts : 0
TLS client handshake failures : 0
TLS server handshake failures : 0

Cisco has a direct path to talk to Microsoft engineering as part of the Cisco Cube being certified for Direct Routing. If the community suggests that logging changes need to be made to reduce alarms I would suggest raising this to the Cisco account managers for your account. They should be able to engage that team internally and either suggest documentation changes to the configuration guide or have Cisco engage with Microsoft engineering to look into anything.

oeschger01 · ‎06-13-2023

Hello Greg,
thank you for your valuable input an your informative link.

Do I understand you correctly that the observed TLS closures take place because Microsoft does not regard CUBE's sip-options keepalive messages to reset its TLS timers and because CUBE does not support RFC 5626 (which Microsoft uses to reset its TLS timers)? In other words, would the problem with the TLS closures be solved if CUBE supported RFC 5626?

Direct Routing for Microsoft Phone System with Cisco Unified Border Element