CUBE Outbound Calling Problems With Auth to ITSP

[Mike Wagner]

Hello,

We have a 2911 CUBE that was configured to use NexVortex for a SIP ITSP, and has worked for years.  Suddenly, outbound calls ceased to work.  Inbound calling works fine.  NXVX requires authentication in the SIP invite.  What's happening is the initial invite is sent by the CUBE, NXVX sends a 407 saying authentication is required, and then the CUBE sends an ACK, that's it.  No Invite with authentication creds.  Now, what's interesting is, on the CUBE when I do a debug ccsip messages, it shows the authenticated invite, which I'll paste below along with relevant bits of code.

Any help is greatly appreciated.  We have nearly 20 man hours in this, and it's becoming a huge problem for us.

SIP Config:

voice service voip
ip address trusted list
ipv4 10.70.21.10 255.255.255.255
ipv4 66.23.129.253 255.255.255.255
ipv4 66.23.138.162 255.255.255.255
ipv4 66.23.190.100 255.255.255.255
ipv4 66.23.190.200 255.255.255.255
ipv4 104.219.163.73 255.255.255.255
ipv4 104.219.163.84 255.255.255.255
ipv4 104.219.163.85 255.255.255.255
ipv4 104.219.162.21 255.255.255.255
ipv4 104.219.162.22 255.255.255.255
ipv4 104.219.162.23 255.255.255.255
ipv4 209.193.79.80 255.255.255.255
ipv4 10.14.160.0 255.255.255.0
address-hiding
mode border-element license capacity 200
allow-connections sip to sip
redirect ip2ip
fax protocol t38 version 0 ls-redundancy 0 hs-redundancy 0 fallback none
modem passthrough nse codec g711ulaw
sip
rel1xx disable
asserted-id pai
localhost dns:px1.nexvortex.com
early-offer forced
midcall-signaling passthru
privacy-policy passthru
no call service stop

sip-ua
credentials username kJ13580bVLR password 7 XXXXXXXX realm nexvortex.com
authentication username kJ13580bVLR password 7 XXXXXXXX realm nexvortex.com
retry invite 2
retry register 10
timers connect 100
registrar dns:px1.nexvortex.com expires 3600
sip-server dns:px1.nexvortex.com
connection-reuse via-port

Outbound Dial Peer:

dial-peer voice 101 voip
translation-profile outgoing StripTechPrefix
destination-pattern 1#.T
session protocol sipv2
session target sip-server
incoming uri via SIP
voice-class codec 1
voice-class sip early-offer forced
voice-class sip bind control source-interface GigabitEthernet0/0
voice-class sip bind media source-interface GigabitEthernet0/0
dtmf-relay rtp-nte
no vad

Attached is the debug ccsip messages from the call from "4196735555" to "9375551212"  Also, "1.1.1.1" is the static external IP of Gig0/0, assigned directly to it.

Thank you in advance!!!

[Mike Wagner]

I wanted to share that, after receiving the 407, the CUBE sends an ACK, then NexVortex receives nothing else.  Debug ccsip messages shows another invite from the CUBE containing actual registration info. But when I do a PCAP from the upstream switch, I never see that packet.  What would make it show in debug ccsip messages but not actually be sent?

[b.winter]

In the trace, CUBE clearly sends the INVITE with Auth 3 times and after it didn't get any reply, the request times out. So from this point of view, CUBE does everything correctly.

If you say, that the message is not going through the correct interface, then you might have a config or routing issue.
Have you captured the traffic on the CUBE's interface, which is used to communicate with the provider?

And for info:
You shouldn't use a SIP trunk configuration in SIP-UA section anymore. The "standard" today is using Tenants. You can check out the following doc as an example https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/voice/cube/ios-xe/config/ios-xe-book/m_voi-cube-multi-tenants.html

[Roger Kallberg]

Not directly related to you issue as such, but from the look of your shared configuration your using dial peer 101 both in the outbound and inbound direction. This could work, but is IMHO not the best to do. It would be better if you where to split these into separate once. To do so you could do this.

dial-peer voice 101 voip
 no incoming uri via SIP
!
dial-peer voice 100 voip
 description Incoming Dial Peer from ITSP
 session protocol sipv2
 incoming uri via SIP
 voice-class codec 1  
 voice-class sip bind control source-interface GigabitEthernet0/0
 voice-class sip bind media source-interface GigabitEthernet0/0
 dtmf-relay rtp-nte
 no vad

As @b.winter stated you are sending the authentication info to the ITSP, but if they claim that they don't get it I would start looking at routing or configuration issues. Would it be possible for you to share the output of these two debugs running at the same time, debug ccsip message and debug voip ccapi inout? It would also be helpful if you could share your entire running configuration so that we can verify it.

[b.winter]

@Mike Wagner Any update on here?

[Mike Wagner]

Thanks All!  I'm rating your suggestions/answers as helpful, as they are definitely pertinent to resolving this issue in most cases.  Our case turned out to be network related.  And it was incredibly strange and hard to diagnose.  I worked on it til 1 AM the night before Thanksgiving.

We recently refreshed our Cisco Firepower firewalls with newer models.  Same configs, platform policies, NAT policies, etc.  This will become relevant later...

The UDP Invite packets from the CUBE were so large that they were being sent as UDP fragments.  Wireshark on my system wasn't reassembling them from the ERSPAN.  It simply said fragmented UDP datagram.  So, I thought they were just noise or unimportant to the issue.  Once I was desperate enough to investigate them further and find a way to reassemble them, I realized they were the missing auth packets.  The new Firepower 3140 was rejecting them.  The previous Firepower 4110 had no issues with them.  There are some changes in the core platform and FXOS/FTD integration from the 4100 series to the 3100 series.  The packets must have been reassembled by FXOS on the 4100 before they reached the FTD instance.  On the 3100, it was outright rejecting them upon receipt.  So, I adjusted the interface policy to allow fragmented packets, and now all is well.

Whew!  That is one I don't want to relive!