This document was generated from CDN thread
Created by: Jason Jackson on 20-03-2012 09:27:55 AM
Hello everyone.
First off, I'm far from a developer. I'm just the guy that manages the hardware/ios on the non-native components in a comprehensive CVP model. That includes things like ingress, CUSP, and vxml browsers. The developers have asked me to figure out why a percentage of calls fail with a "event=error.noresource".
Our vxml browsers are version c5350-is-mz.151-2.T1.bin. I've been running the following debugs to try and capture the issue:
MRCP:
MRCP client session debugging is on
MRCP client error debugging is on
VXML:
debug voip application vxml error software is ON
debug voip application vxml error call is ON (filter is OFF)
debug voip application vxml application is ON (filter is OFF)
debug voip application vxml inout is ON (filter is OFF)
debug voip application vxml event is ON (filter is OFF)
debug voip application vxml trace is ON (filter is OFF)
debug voip application vxml log is ON (filter is OFF)
debug voip application vxml warning is ON (filter is OFF)
debug voip application vxml background is ON (filter is OFF)
debug voip application vxml gtd is ON (filter is OFF)
HTTP Client:
HTTP Client Messages debugging is on
We can't seem to reproduce this issue in our labs and it's been quite a pain to debug in production. I've had to weight one of our 25 vxml browsers very low in CUSP so that it's only take a handful of calls. Even with a few calls the CPU is really hit hard with the debugs I'm running.
I've been able to capture a few key items from the debugs and thought I would share them here while I wait for TAC to get on the same page with me.
****************Error below shows some sort of failure from our backend ASR***************
Mar 19 18:52:45 [] 80869597: .Mar 19 13:53:07.886 CDT: //-1//MRCP:/mrcp_response_handler:
Mar 19 18:52:45 [] 80869598: status=RTSPLIB_STATUS_SOCKET_ERROR
Mar 19 18:52:45 [] 80869599: .Mar 19 13:53:07.886 CDT: //-1//MRCP:/delete_from_hoststatus_table:
Mar 19 18:52:45 [] 80869600: Ignoring current failure from (10.175.230.86:554) Current no failures(1)
Mar 19 18:52:45 [] 80869601: .Mar 19 13:53:07.886 CDT: //13252772//MRCP:/mrcp_send_recog_app_response:
Mar 19 13:53:07.886 CDT: //-1//MRCP:/mrcp_reset_fsm_after_failure:
*****************A few lines later in the debug I see the no resource*************************
.Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_jse_global_switch:
Mar 19 11:47:43 [] 59269517: switch to scope(dialog)
Mar 19 11:47:43 [] 59269518: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_expr_eval:
Mar 19 11:47:43 [] 59269519: expr=(var _message=undefined)
Mar 19 11:47:43 [] 59269520: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_catch_proc:
Mar 19 11:47:43 [] 59269521: <catch>:
Mar 19 11:47:43 [] 59269522: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_nmtokens_proc:
Mar 19 11:47:43 [] 59269523: name=error.noresource count=1
Mar 19 11:47:43 [] 59269524: script_loading=0
Mar 19 11:47:43 [] 59269525: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_asynceventparams_get:
Mar 19 11:47:43 [] 59269526:
Mar 19 11:47:43 [] 59269527: .Mar 19 06:48:06.386 CDT: vxml_asynceventparams_get:
Mar 19 11:47:43 [] 59269528: <var>: namep=audium_hotevent expr='NoResource'
Mar 19 11:47:43 [] 59269529: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_expr_eval:
Mar 19 11:47:43 [] 59269530: expr=(var audium_hotevent='NoResource')
Mar 19 11:47:43 [] 59269531: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_jse_global_switch:
Mar 19 11:47:43 [] 59269532: switch to scope(anonymous)
Mar 19 11:47:43 [] 59269533: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_submit_proc:
Mar 19 11:47:43 [] 59269534: <submit>:
**********************************************************************************************
Any ideas? I'm assuming that failure kicks off the noresource issue but Nuance has been unable to find anything on the backend. TAC is taking awhile to even understand the issue as well. Thought I would post here and see what other folks had to say about it.
Thanks!
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Janine Graves on 20-03-2012 10:17:37 AM
Hi,
In the classroom, error.noresource is the 'generic' (not at all useful) error that the VoiceXML gateway returns when there's any kind of ASR problem with a grammar. It also occurs if the gateway times out waiting for an HTTP response (Media server slow to return a value, Vxml Server slow to return the next VXML page).
In your log, it looks like perhaps the gateway is timing out waiting for a reponse or a connection to the ASR resource.
Have you tried changing the VXML Gateway CLI to increase the MRCP timeouts, or the HTTP timeouts?
This is the amount of time the gateway waits for a response from the ASR:
mrcp client timeout connect 10
mrcp client timeout message 10
I've seen this occur in class during app development if the ASR can't retrieve (or times out retrieving) an external grammar, of if the grammar it retrieves is not valid.
Since this only happens occasionally, I'd guess it's a timeout of some sort (or possibly a load issue on the gateway).
It would be good to know whether this always occurs at the same place in the Studio application (ie, there's a bad or missing grammar file), or if it's random.
Do you know if the Studio application is changing the 'fetchtimeout' timer? This timer tells the gateway how long to wait for the next document to be retrieved.
You can search (using Notepad++) the VXMLServer/applications/<appname>/ folder for the word 'fetchtimeout'.
If it's not set in the application , then you CAN change the response timeout to a larger value in the gateway CLI (unfortunately, this will be overridden by any settings done in the Studio application):
http client connection timeout 10
http client response timeout 10
There might also be a way to configure the ASR to wait a little longer if it's loading an external grammar. Something similar to the 'fetchtimeout' timer. But, if you're not seeing any errors in your MRCP logs (on the ASR server), then the error's probably on the gateway side of things.
Janine
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Jason Jackson on 21-03-2012 03:05:20 PM
Hi Janine - thanks for the response.
The mrcp timers and http timers are all set as you describe and the fetchtimout doesn't seem to be the issue.
I can tell you that we have now been doing some deep packet captures on the vxml browser and it seems that when this error is thrown the vxml browser sees RTSP response come back from our ASR with "Session Not Found". The message right before that the browser is doing an RTSP ANNOUNCE. A few lines later in the capture we see the no.resource.
Cisco BU has suggested that Nuance change the rtsp timer from 300 seconds (what it's currently set at) to -1 (unlimited). We are in the process of setting up a browser to point to one nuance box on the backend so we can at least prove this out but it sounds good right?
Thoughts?
Subject: Re: New Message from Jason Jackson in Customer Voice Portal (CVP) - CVP - A
Replied by: Janine Graves on 21-03-2012 03:33:16 PM
That sounds good. Keep us posted as to the result! On 3/21/2012 3:05 PM, Cisco Developer Community Forums wrote: Jason Jackson has created a new message in the forum "CVP - All Versions": -------------------------------------------------------------- Hi Janine - thanks for the response. The mrcp timers and http timers are all set as you describe and the fetchtimout doesn't seem to be the issue. I can tell you that we have now been doing some deep packet captures on the vxml browser and it seems that when this error is thrown the vxml browser sees RTSP response come back from our ASR with "Session Not Found". The message right before that the browser is doing an RTSP ANNOUNCE. A few lines later in the capture we see the no.resource. Cisco BU has suggested that Nuance change the rtsp timer from 300 seconds (what it's currently set at) to -1 (unlimited). We are in the process of setting up a browser to point to one nuance box on the backend so we can at least prove this out but it sounds good right? Thoughts? -- To respond to this post, please click the following link: <http://developer.cisco.com/web/cvp/forums/-/message_boards/view_message/5334656> or simply reply to this email. -- Janine Graves
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Shikha Sharma on 23-03-2012 05:27:03 PM
We recently encountered this issue and the root cause was licensing problem on Nuance server but then our application was using TTS extensively and the call volume was also very high. I would run the licensing tool and monitor logs under "<>//Nuance/license/logs" on Nuance server to check on available licenses for ASR/TTS when the issue is encountered.
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Jason Jackson on 27-03-2012 02:45:54 PM
So we think we have multiple issues.
1) We have confirmed that the rtsp timer session does in fact expire on some of our calls and will need to be bumped up on the Nuance side.
2) VXML browser/Nuance - CSEQ issue (see below)
With further debugs/packet captures we can see that there is a mix up on the browser with the CSEQ headers. Nuance sends it over but the browser doesn't parse it properly.
Here is a snippet from the debugs on the browser:
T: //-1//RTSP:/rtsp_pmh_parse_svr_response:
Mar 27 14:45:40 [10.173.93.30] 248154156: .Mar 27 09:45:53.756 CDT: ########################################
Mar 27 14:45:40 [10.173.93.30] 248154157: .Mar 27 09:45:53.756 CDT: Mesg_line :RTSP/1.0 200 OK
Mar 27 14:45:40 [10.173.93.30] 248154158: .Mar 27 09:45:53.756 CDT: Content_length :32
Mar 27 14:45:40 [10.173.93.30] 248154159: .Mar 27 09:45:53.756 CDT: Header list
Mar 27 14:45:40 [10.173.93.30] 248154160: .Mar 27 09:45:53.756 CDT: Content-Length: 32
Mar 27 14:45:40 [10.173.93.30] 248154161: .Mar 27 09:45:53.756 CDT: Content-Type: application/mrcp
Mar 27 14:45:40 [10.173.93.30] 248154162: .Mar 27 09:45:53.756 CDT: Session: EPHBNCGJAAAHNGGOAAAAAACB
Mar 27 14:45:40 [10.173.93.30] 248154163: .Mar 27 09:45:53.756 CDT: Message Body
Mar 27 14:45:40 [10.173.93.30] 248154164: .Mar 27 09:45:53.756 CDT: MRCP/1.0 107 200 IN-PROGRESS
Mar 27 14:45:40 [10.173.93.30] 248154165:
Mar 27 14:45:40 [10.173.93.30] 248154166: .Mar 27 09:45:53.756 CDT: ########################################
Mar 27 14:45:40 [10.173.93.30] 248154167: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_process_single_svr_resp: complete
Mar 27 14:45:40 [10.173.93.30] 248154168: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_process_response:
Mar 27 14:45:40 [10.173.93.30] 248154169: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_process_response: No CSeq
Any ideas? The packet capture proves that there is a CSEQ sent in the last IN-PROGRESS message but the browser refuses to parse it properly. Anyone ever heard of this type of issue?
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Jason Jackson on 13-04-2012 10:48:45 AM
Quick update on this...
- changing the RTSP timer issue on Nuance did help the issue
-I helped TAC identify a CSEQ issue where the browser ignores a CSeQ in the RTSP/MRCP traffic if recieved in a very specific way. No bug ID documnted but if you ever see this in the logs you will know you're hitting the issue:
Mar 27 14:45:40 [10.173.93.30] 248154144: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_process_single_svr_resp:
Mar 27 14:45:40 [10.173.93.30] 248154145: .Mar 27 09:45:53.756 CDT: rtsp_process_single_svr_resp: 400 bytes of data:
Mar 27 14:45:40 [10.173.93.30] 248154146: CSeq: 99 Mar 27 14:45:40 [10.173.93.30] 248154147: Session: EPHBNCGJAAAHNGGOAAAAAACB
Mar 27 14:45:40 [10.173.93.30] 248154148: Content-Type: application/mrcp
Mar 27 14:45:40 [10.173.93.30] 248154149: Content-Length: 32
Mar 27 14:45:40 [10.173.93.30] 248154150:
Mar 27 14:45:40 [10.173.93.30] 248154151: MRCP/1.0 107 200 IN-PROGRESS
Mar 27 14:45:40 [10.173.93.30] 248154152:
Mar 27 14:45:40 [10.173.93.30] 248154153: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_parse_svr_response:
Mar 27 14:45:40 [10.173.93.30] 248154154: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_parse_svr_response: just one response(may be partial): 32
Mar 27 14:45:40 [10.173.93.30] 248154155: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_parse_svr_response:
Mar 27 14:45:40 [10.173.93.30] 248154156: .Mar 27 09:45:53.756 CDT: ########################################
Mar 27 14:45:40 [10.173.93.30] 248154157: .Mar 27 09:45:53.756 CDT: Mesg_line :RTSP/1.0 200 OK
Mar 27 14:45:40 [10.173.93.30] 248154158: .Mar 27 09:45:53.756 CDT: Content_length :32
Mar 27 14:45:40 [10.173.93.30] 248154159: .Mar 27 09:45:53.756 CDT: Header list
Mar 27 14:45:40 [10.173.93.30] 248154160: .Mar 27 09:45:53.756 CDT: Content-Length: 32
Mar 27 14:45:40 [10.173.93.30] 248154161: .Mar 27 09:45:53.756 CDT: Content-Type: application/mrcp
Mar 27 14:45:40 [10.173.93.30] 248154162: .Mar 27 09:45:53.756 CDT: Session: EPHBNCGJAAAHNGGOAAAAAACB
Mar 27 14:45:40 [10.173.93.30] 248154163: .Mar 27 09:45:53.756 CDT: Message Body
Mar 27 14:45:40 [10.173.93.30] 248154164: .Mar 27 09:45:53.756 CDT: MRCP/1.0 107 200 IN-PROGRESS
Mar 27 14:45:40 [10.173.93.30] 248154165:
Mar 27 14:45:40 [10.173.93.30] 248154166: .Mar 27 09:45:53.756 CDT: ########################################
Mar 27 14:45:40 [10.173.93.30] 248154167: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_process_single_svr_resp: complete
Mar 27 14:45:40 [10.173.93.30] 248154168: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_process_response:
Mar 27 14:45:40 [10.173.93.30] 248154169: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_process_response: No CSeq
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Jason Jackson on 10-07-2012 01:21:48 PM
Probably my last update on this but here is the bug ID for the no resource conditions:
CSCua14855 Bug ID:
VXML gateway is not parsing MRCP CSEQ header correctly
Symptom:
Issue when parsing received RTSP messages, when the RTSP/1.0 200 OK
is separated from the rest of the message, and the first part of the
message is cut on the line boundary (complete line).
the RTSP error debug log shows an error of "No CSeq".
Conditions:
The symptom is observed with MRCP is enabled.
This issue was noticed for RTSP sockets active for longer than 5 minutes.
Workaround:
None
Hopefully this will help other folks. I know it gave us a headache for a long time. We are currently running a special IOS image until the code is fixed in a main line release.
Created by: Jason Jackson on 20-03-2012 09:27:55 AM
Hello everyone.
First off, I'm far from a developer. I'm just the guy that manages the hardware/ios on the non-native components in a comprehensive CVP model. That includes things like ingress, CUSP, and vxml browsers. The developers have asked me to figure out why a percentage of calls fail with a "event=error.noresource".
Our vxml browsers are version c5350-is-mz.151-2.T1.bin. I've been running the following debugs to try and capture the issue:
MRCP:
MRCP client session debugging is on
MRCP client error debugging is on
VXML:
debug voip application vxml error software is ON
debug voip application vxml error call is ON (filter is OFF)
debug voip application vxml application is ON (filter is OFF)
debug voip application vxml inout is ON (filter is OFF)
debug voip application vxml event is ON (filter is OFF)
debug voip application vxml trace is ON (filter is OFF)
debug voip application vxml log is ON (filter is OFF)
debug voip application vxml warning is ON (filter is OFF)
debug voip application vxml background is ON (filter is OFF)
debug voip application vxml gtd is ON (filter is OFF)
HTTP Client:
HTTP Client Messages debugging is on
We can't seem to reproduce this issue in our labs and it's been quite a pain to debug in production. I've had to weight one of our 25 vxml browsers very low in CUSP so that it's only take a handful of calls. Even with a few calls the CPU is really hit hard with the debugs I'm running.
I've been able to capture a few key items from the debugs and thought I would share them here while I wait for TAC to get on the same page with me.
****************Error below shows some sort of failure from our backend ASR***************
Mar 19 18:52:45 [] 80869597: .Mar 19 13:53:07.886 CDT: //-1//MRCP:/mrcp_response_handler:
Mar 19 18:52:45 [] 80869598: status=RTSPLIB_STATUS_SOCKET_ERROR
Mar 19 18:52:45 [] 80869599: .Mar 19 13:53:07.886 CDT: //-1//MRCP:/delete_from_hoststatus_table:
Mar 19 18:52:45 [] 80869600: Ignoring current failure from (10.175.230.86:554) Current no failures(1)
Mar 19 18:52:45 [] 80869601: .Mar 19 13:53:07.886 CDT: //13252772//MRCP:/mrcp_send_recog_app_response:
Mar 19 13:53:07.886 CDT: //-1//MRCP:/mrcp_reset_fsm_after_failure:
*****************A few lines later in the debug I see the no resource*************************
.Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_jse_global_switch:
Mar 19 11:47:43 [] 59269517: switch to scope(dialog)
Mar 19 11:47:43 [] 59269518: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_expr_eval:
Mar 19 11:47:43 [] 59269519: expr=(var _message=undefined)
Mar 19 11:47:43 [] 59269520: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_catch_proc:
Mar 19 11:47:43 [] 59269521: <catch>:
Mar 19 11:47:43 [] 59269522: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_nmtokens_proc:
Mar 19 11:47:43 [] 59269523: name=error.noresource count=1
Mar 19 11:47:43 [] 59269524: script_loading=0
Mar 19 11:47:43 [] 59269525: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_asynceventparams_get:
Mar 19 11:47:43 [] 59269526:
Mar 19 11:47:43 [] 59269527: .Mar 19 06:48:06.386 CDT: vxml_asynceventparams_get:
Mar 19 11:47:43 [] 59269528: <var>: namep=audium_hotevent expr='NoResource'
Mar 19 11:47:43 [] 59269529: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_expr_eval:
Mar 19 11:47:43 [] 59269530: expr=(var audium_hotevent='NoResource')
Mar 19 11:47:43 [] 59269531: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_jse_global_switch:
Mar 19 11:47:43 [] 59269532: switch to scope(anonymous)
Mar 19 11:47:43 [] 59269533: .Mar 19 06:48:06.386 CDT: //13242691/0C0866BFBECB/VXML:/vxml_submit_proc:
Mar 19 11:47:43 [] 59269534: <submit>:
**********************************************************************************************
Any ideas? I'm assuming that failure kicks off the noresource issue but Nuance has been unable to find anything on the backend. TAC is taking awhile to even understand the issue as well. Thought I would post here and see what other folks had to say about it.
Thanks!
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Janine Graves on 20-03-2012 10:17:37 AM
Hi,
In the classroom, error.noresource is the 'generic' (not at all useful) error that the VoiceXML gateway returns when there's any kind of ASR problem with a grammar. It also occurs if the gateway times out waiting for an HTTP response (Media server slow to return a value, Vxml Server slow to return the next VXML page).
In your log, it looks like perhaps the gateway is timing out waiting for a reponse or a connection to the ASR resource.
Have you tried changing the VXML Gateway CLI to increase the MRCP timeouts, or the HTTP timeouts?
This is the amount of time the gateway waits for a response from the ASR:
mrcp client timeout connect 10
mrcp client timeout message 10
I've seen this occur in class during app development if the ASR can't retrieve (or times out retrieving) an external grammar, of if the grammar it retrieves is not valid.
Since this only happens occasionally, I'd guess it's a timeout of some sort (or possibly a load issue on the gateway).
It would be good to know whether this always occurs at the same place in the Studio application (ie, there's a bad or missing grammar file), or if it's random.
Do you know if the Studio application is changing the 'fetchtimeout' timer? This timer tells the gateway how long to wait for the next document to be retrieved.
You can search (using Notepad++) the VXMLServer/applications/<appname>/ folder for the word 'fetchtimeout'.
If it's not set in the application , then you CAN change the response timeout to a larger value in the gateway CLI (unfortunately, this will be overridden by any settings done in the Studio application):
http client connection timeout 10
http client response timeout 10
There might also be a way to configure the ASR to wait a little longer if it's loading an external grammar. Something similar to the 'fetchtimeout' timer. But, if you're not seeing any errors in your MRCP logs (on the ASR server), then the error's probably on the gateway side of things.
Janine
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Jason Jackson on 21-03-2012 03:05:20 PM
Hi Janine - thanks for the response.
The mrcp timers and http timers are all set as you describe and the fetchtimout doesn't seem to be the issue.
I can tell you that we have now been doing some deep packet captures on the vxml browser and it seems that when this error is thrown the vxml browser sees RTSP response come back from our ASR with "Session Not Found". The message right before that the browser is doing an RTSP ANNOUNCE. A few lines later in the capture we see the no.resource.
Cisco BU has suggested that Nuance change the rtsp timer from 300 seconds (what it's currently set at) to -1 (unlimited). We are in the process of setting up a browser to point to one nuance box on the backend so we can at least prove this out but it sounds good right?
Thoughts?
Subject: Re: New Message from Jason Jackson in Customer Voice Portal (CVP) - CVP - A
Replied by: Janine Graves on 21-03-2012 03:33:16 PM
That sounds good. Keep us posted as to the result! On 3/21/2012 3:05 PM, Cisco Developer Community Forums wrote: Jason Jackson has created a new message in the forum "CVP - All Versions": -------------------------------------------------------------- Hi Janine - thanks for the response. The mrcp timers and http timers are all set as you describe and the fetchtimout doesn't seem to be the issue. I can tell you that we have now been doing some deep packet captures on the vxml browser and it seems that when this error is thrown the vxml browser sees RTSP response come back from our ASR with "Session Not Found". The message right before that the browser is doing an RTSP ANNOUNCE. A few lines later in the capture we see the no.resource. Cisco BU has suggested that Nuance change the rtsp timer from 300 seconds (what it's currently set at) to -1 (unlimited). We are in the process of setting up a browser to point to one nuance box on the backend so we can at least prove this out but it sounds good right?
Thoughts? -- To respond to this post, please click the following link: <http://developer.cisco.com/web/cvp/forums/-/message_boards/view_message/5334656> or simply reply to this email. -- Janine GravesSubject: RE: No Resource Failures CVP/VXML Browser
Replied by: Shikha Sharma on 23-03-2012 05:27:03 PM
We recently encountered this issue and the root cause was licensing problem on Nuance server but then our application was using TTS extensively and the call volume was also very high. I would run the licensing tool and monitor logs under "<>//Nuance/license/logs" on Nuance server to check on available licenses for ASR/TTS when the issue is encountered.
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Jason Jackson on 27-03-2012 02:45:54 PM
So we think we have multiple issues.
1) We have confirmed that the rtsp timer session does in fact expire on some of our calls and will need to be bumped up on the Nuance side.
2) VXML browser/Nuance - CSEQ issue (see below)
With further debugs/packet captures we can see that there is a mix up on the browser with the CSEQ headers. Nuance sends it over but the browser doesn't parse it properly.
Here is a snippet from the debugs on the browser:
T: //-1//RTSP:/rtsp_pmh_parse_svr_response:
Mar 27 14:45:40 [10.173.93.30] 248154156: .Mar 27 09:45:53.756 CDT: ########################################
Mar 27 14:45:40 [10.173.93.30] 248154157: .Mar 27 09:45:53.756 CDT: Mesg_line :RTSP/1.0 200 OK
Mar 27 14:45:40 [10.173.93.30] 248154158: .Mar 27 09:45:53.756 CDT: Content_length :32
Mar 27 14:45:40 [10.173.93.30] 248154159: .Mar 27 09:45:53.756 CDT: Header list
Mar 27 14:45:40 [10.173.93.30] 248154160: .Mar 27 09:45:53.756 CDT: Content-Length: 32
Mar 27 14:45:40 [10.173.93.30] 248154161: .Mar 27 09:45:53.756 CDT: Content-Type: application/mrcp
Mar 27 14:45:40 [10.173.93.30] 248154162: .Mar 27 09:45:53.756 CDT: Session: EPHBNCGJAAAHNGGOAAAAAACB
Mar 27 14:45:40 [10.173.93.30] 248154163: .Mar 27 09:45:53.756 CDT: Message Body
Mar 27 14:45:40 [10.173.93.30] 248154164: .Mar 27 09:45:53.756 CDT: MRCP/1.0 107 200 IN-PROGRESS
Mar 27 14:45:40 [10.173.93.30] 248154165:
Mar 27 14:45:40 [10.173.93.30] 248154166: .Mar 27 09:45:53.756 CDT: ########################################
Mar 27 14:45:40 [10.173.93.30] 248154167: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_process_single_svr_resp: complete
Mar 27 14:45:40 [10.173.93.30] 248154168: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_process_response:
Mar 27 14:45:40 [10.173.93.30] 248154169: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_process_response: No CSeq
Any ideas? The packet capture proves that there is a CSEQ sent in the last IN-PROGRESS message but the browser refuses to parse it properly. Anyone ever heard of this type of issue?
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Jason Jackson on 13-04-2012 10:48:45 AM
Quick update on this...
- changing the RTSP timer issue on Nuance did help the issue
-I helped TAC identify a CSEQ issue where the browser ignores a CSeQ in the RTSP/MRCP traffic if recieved in a very specific way. No bug ID documnted but if you ever see this in the logs you will know you're hitting the issue:
Mar 27 14:45:40 [10.173.93.30] 248154144: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_process_single_svr_resp:
Mar 27 14:45:40 [10.173.93.30] 248154145: .Mar 27 09:45:53.756 CDT: rtsp_process_single_svr_resp: 400 bytes of data:
Mar 27 14:45:40 [10.173.93.30] 248154146: CSeq: 99 Mar 27 14:45:40 [10.173.93.30] 248154147: Session: EPHBNCGJAAAHNGGOAAAAAACB
Mar 27 14:45:40 [10.173.93.30] 248154148: Content-Type: application/mrcp
Mar 27 14:45:40 [10.173.93.30] 248154149: Content-Length: 32
Mar 27 14:45:40 [10.173.93.30] 248154150:
Mar 27 14:45:40 [10.173.93.30] 248154151: MRCP/1.0 107 200 IN-PROGRESS
Mar 27 14:45:40 [10.173.93.30] 248154152:
Mar 27 14:45:40 [10.173.93.30] 248154153: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_parse_svr_response:
Mar 27 14:45:40 [10.173.93.30] 248154154: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_parse_svr_response: just one response(may be partial): 32
Mar 27 14:45:40 [10.173.93.30] 248154155: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_parse_svr_response:
Mar 27 14:45:40 [10.173.93.30] 248154156: .Mar 27 09:45:53.756 CDT: ########################################
Mar 27 14:45:40 [10.173.93.30] 248154157: .Mar 27 09:45:53.756 CDT: Mesg_line :RTSP/1.0 200 OK
Mar 27 14:45:40 [10.173.93.30] 248154158: .Mar 27 09:45:53.756 CDT: Content_length :32
Mar 27 14:45:40 [10.173.93.30] 248154159: .Mar 27 09:45:53.756 CDT: Header list
Mar 27 14:45:40 [10.173.93.30] 248154160: .Mar 27 09:45:53.756 CDT: Content-Length: 32
Mar 27 14:45:40 [10.173.93.30] 248154161: .Mar 27 09:45:53.756 CDT: Content-Type: application/mrcp
Mar 27 14:45:40 [10.173.93.30] 248154162: .Mar 27 09:45:53.756 CDT: Session: EPHBNCGJAAAHNGGOAAAAAACB
Mar 27 14:45:40 [10.173.93.30] 248154163: .Mar 27 09:45:53.756 CDT: Message Body
Mar 27 14:45:40 [10.173.93.30] 248154164: .Mar 27 09:45:53.756 CDT: MRCP/1.0 107 200 IN-PROGRESS
Mar 27 14:45:40 [10.173.93.30] 248154165:
Mar 27 14:45:40 [10.173.93.30] 248154166: .Mar 27 09:45:53.756 CDT: ########################################
Mar 27 14:45:40 [10.173.93.30] 248154167: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_process_single_svr_resp: complete
Mar 27 14:45:40 [10.173.93.30] 248154168: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_process_response:
Mar 27 14:45:40 [10.173.93.30] 248154169: .Mar 27 09:45:53.756 CDT: //-1//RTSP:/rtsp_pmh_process_response: No CSeq
Subject: RE: No Resource Failures CVP/VXML Browser
Replied by: Jason Jackson on 10-07-2012 01:21:48 PM
Probably my last update on this but here is the bug ID for the no resource conditions:
CSCua14855 Bug ID:
VXML gateway is not parsing MRCP CSEQ header correctly
Symptom:
Issue when parsing received RTSP messages, when the RTSP/1.0 200 OK
is separated from the rest of the message, and the first part of the
message is cut on the line boundary (complete line).
the RTSP error debug log shows an error of "No CSeq".
Conditions:
The symptom is observed with MRCP is enabled.
This issue was noticed for RTSP sockets active for longer than 5 minutes.
Workaround:
None
Hopefully this will help other folks. I know it gave us a headache for a long time. We are currently running a special IOS image until the code is fixed in a main line release.
