CVP 11.6 CallServer no response to SIP INVITE

[clearbluelou]

Has anyone ever seen an issue in which the CVP CallServer suddenly stops responding to SIP INVITEs?  Meaning, one moment it was responding properly with 100 Trying and 200 OK and the next it stopped responding altogether.  There is no record whatsoever of the INVITE in the CallServer logs (c:\cisco\cvp\logs\).  A packet capture shows the INVITE was properly delivered to the CallServer's IP address.  Neither of the Windows Event Viewer logs, application and system, show any errors or issues of any kind.  The CallServer's tomcat8.exe process remained running.  As far as I can determine, nothing was blocking TCP/5060.  Then, for lack of any better ideas, I restarted the CallServer service and it started properly responding to SIP INVITEs again.  This has happened at random times and on random days about four times over the past six months.  I have no idea what else to look for.  Oh, and I should mention that we have a redundant data center with exactly the same set up, version, patches, etc. in which this has never happened.

[bill.king1]

You mentioned CVP, but what about VXML apps/logs? Is there any correlation with that?
So basically everything is good until you totally go in the mud?

Do existing calls drop or is it more of a case where all new calls fail?

[clearbluelou]

Thank you for your reply.  Unfortunately, since our CVP uses the Comprehensive model, it depends on the CallServer to get instructions from ICM before the VXML applications are engaged.  Since the CallServer never received the calls, there was no way to launch VXML.

As to whether or not existing calls drop, that's a good question.  I had not thought to check.  The CallServer logs make it look like existing calls were able to complete the VXML apps and the CallServer sent SIP REFER messages for the transfers.  However, our SBC logs don't show those same REFER messages being received.  This tells me that something on the network between the CVP server and the SBC likely blocked all communication.

[bill.king1]

Right, what I was getting at was more, when this issue happens, what do you see in the VXML logs. Like is it a case where the whole server is hosed/unresponsive due to some unknown issue, which is manifesting itself as a CVP issue, but is really a server type issue?

Or is it more like the other scenario outlined below, where it is more of a network issue?

[clearbluelou]

Thanks for the clarification.  I was on vacation last week so the VXML log data is long gone now.  I will keep this in mind if/when the issue happens again.

[clearbluelou]

So, this issue has happened a few more times since my last update.  I'm still working with our NetOps team and they are primarily focused on our Palo Alto firewall.  We have found some issues in which other SIP sessions go "stale" on the firewall and the OPTIONS messages from both sides stop getting responses.  I think that is what is causing the issue with CVP but we're not 100% sure yet.

Interestingly, I have verified that all of the existing SIP sessions continue unaffected (i.e. the VXML apps complete cleanly) until the customer transfers out of the IVR (using SIP REFER).  Only the SIP INVITE messages for new calls are redirected because the SBC changes the session agent to out of service when it stops getting 200 OK responses to the OPTIONS messages.

[bill.king1]

This may be unrelated to your issue, but did you happen to see this defect? Since you mention it happens on the calls transferred out to agents, and is intermittent, I thought you might find this of interest. Are you using whisper?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx64067

[clearbluelou]

Bill,

Thank you for the follow up.  I have reviewed that bug.  I don't think it is related.  The firewall guys tell me that it won't tear down any sessions or mark them stale in less than 24 hours.  Also, neither time frame match the frequency of the issue.

[bill.king1]

Good luck, please let us know what the root call is. There's also this one as well, although not quite again your exact scenario it sounds like.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvx94882

[clearbluelou]

Yeah, I've seen that bug, too.  I think that was the first one I found when this started.  Ironically, our system doesn't exhibit that issue (knock on wood).  All of the IP addresses in the OPTIONS messages are correct.

[Konstantin Vaksin]

Normally, if you get SIP Invite on CVP CS, it should be in the logs.

It might be, you have a problem with Network routing

[clearbluelou]

Thank you for your reply.  Yes, I think that must be the issue.

[Mr Brightside]

At the time of issue, do the packet capture on CVP CS, if you see incoming INVITE as you mentioned thats great, so check for the outgoing SIP message i.e. TRYING or 200, if you have these on the pcap then you must have something in between the ingress gateway and CVP that blocks the response could be your firewall or something.