02-27-2018 09:51 PM - edited 03-17-2019 12:18 PM
Hello,
I have 2 voice gateways with 4 PRIs terminated on each gateway. VG router model is ISR4451-X running isr4400-universalk9.03.16.05.S.155-3.S5-ext.SPA. I am facing issue with DSP alarms/errors causing calls to fail (both incoming an outgoing). I have to reboot the VG router to restore the calling services. Then for 3-4 weeks or sometimes even for couple of months, it won't show any alarms/errors and then suddenly start showing them again. I had raised TAC and they suggested for IOS upgrade which has not resolved the issue. There is one TAC case in progress for the same issue but looks like Cisco engineers are not able to sort out the issue even though I shared all the logs.
I appreciate any help/suggestions on this issue.
Thanks,
Santosh
Here are the logs.
Nov 2 04:43:25.702: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 0035
Nov 2 04:43:25.702: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:43:35.700: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:43:35.700: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 0028
Nov 2 04:43:35.700: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:43:45.704: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:43:45.704: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4568 0028
Nov 2 04:43:45.704: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:43:55.703: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:43:55.703: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 533D 3632
Nov 2 04:43:55.703: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:44:05.702: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:44:05.703: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0033 6E44 0800 4568 0039
Nov 2 04:44:05.703: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:44:15.703: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:44:15.704: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 0038
Nov 2 04:44:15.704: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:44:25.698: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:44:25.698: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 00F8 35ED 0800 4500 0037
Nov 2 04:44:25.699: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:44:34.187: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: bleysen] [Source: 192.168.71.185] [localport: 23] at 04:44:34 GMT Thu Nov 2 2017
Nov 2 04:44:35.701: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:44:35.701: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4568 002B
Nov 2 04:44:35.701: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:44:35.751: %SYS-6-LOGOUT: User bleysen has exited tty session 866(192.168.71.185)
Nov 2 04:44:45.706: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:44:45.706: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 0029
Nov 2 04:44:45.706: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:44:55.700: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:44:55.700: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 0000 0018
Nov 2 04:44:55.700: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:45:05.702: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:45:05.702: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 0037
Nov 2 04:45:05.702: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:45:15.707: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:45:15.707: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4568 002B
Nov 2 04:45:15.707: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:45:25.706: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:45:25.706: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 003A
Nov 2 04:45:25.706: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:45:35.700: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:45:35.700: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4568 0028
Nov 2 04:45:35.700: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:45:44.010: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: crx3hn] [Source: 192.168.68.13] [localport: 23] at 04:45:44 GMT Thu Nov 2 2017
Nov 2 04:45:45.705: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:45:45.705: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 2D1A AC19 4E50 4E56
Nov 2 04:45:45.705: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:45:55.707: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:45:55.707: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 0037
Nov 2 04:45:55.707: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:46:05.701: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:46:05.701: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0007 0006 001C 58F4 7EA0
Nov 2 04:46:05.701: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:46:15.703: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:46:15.703: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4E50 4E56
Nov 2 04:46:15.703: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:46:25.700: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:46:25.700: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 002A 4049 4E50 4E56
Nov 2 04:46:25.700: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:46:35.706: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:46:35.706: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 0028
Nov 2 04:46:35.706: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:46:45.703: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:46:45.703: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 001E FAFF 0800 4500 0028
Nov 2 04:46:45.703: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:46:55.707: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:46:55.707: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 0030
Nov 2 04:46:55.707: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:47:05.699: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:47:05.699: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 000F AC02 050E 0003
Nov 2 04:47:05.699: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:47:15.707: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:47:15.707: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0033 6E44 0800 4568 0039
Nov 2 04:47:15.707: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:47:25.706: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:47:25.706: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 003A
Nov 2 04:47:25.706: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:47:35.708: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:47:35.708: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 001E FAFF 0800 4500 0028
Nov 2 04:47:35.708: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:47:45.700: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:47:45.700: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4500 003A
Nov 2 04:47:45.700: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:47:55.706: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
Nov 2 04:47:55.706: %DSPRM-3-DSPALARMINFO: 0024 0000 0080 0000 000B 0000 4453 5020 4865 6172 7462 6561 7420 5469 6D65 6F75 7421 0000 0000 0000 4568 0034
Nov 2 04:47:55.706: %DSPRM-3-DSPALARMINFO: DSP Heartbeat Timeout!
Nov 2 04:47:57.847: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: crx3hn] [Source: 192.168.68.13] [localport: 23] at 04:47:57 GMT Thu Nov 2 2017
Nov 2 04:48:05.702: %DSPRM-2-DSPALARM: Received alarm indication from dsp (1/1). Resetting the DSP.
INPNVG01#sh voice dsp group all
DSP groups on slot 0/1 slot id 1
dsp 1:
State: UP, firmware: 40.2.11
Max signal/voice channel: 64/64
Max credits: 960, Voice credits: 960, Video credits: 0
num_of_sig_chnls_allocated: 64
Transcoding channels allocated: 0
Group: FLEX_GROUP_VOICE, complexity: FLEX
Shared credits: 945, reserved credits: 0
Signaling channels allocated: 64
Voice channels allocated: 1
Credits used (rounded-up): 15
Voice channels:
Ch01: voice port: 0/1/0:15.1, codec: g711ulaw, credits allocated: 15
Slot: 0/1
Device idx: 0
Dsp Type: SP2700
dsp 2:
State: SOFT_RESET, firmware: 40.2.11
Max signal/voice channel: 64/64
Max credits: 960, Voice credits: 960, Video credits: 0
num_of_sig_chnls_allocated: 56
Transcoding channels allocated: 0
Group: FLEX_GROUP_VOICE, complexity: FLEX
Shared credits: 960, reserved credits: 0
Signaling channels allocated: 56
Voice channels allocated: 0
Credits used (rounded-up): 0
Slot: 0/1
Device idx: 0
Dsp Type: SP2700
DSP groups on slot 0/2 slot id 2
dsp 1:
State: UP, firmware: 40.2.11
Max signal/voice channel: 8/8
Max credits: 120, Voice credits: 120, Video credits: 0
num_of_sig_chnls_allocated: 4
Transcoding channels allocated: 0
Group: FLEX_GROUP_VOICE, complexity: FLEX
Shared credits: 120, reserved credits: 0
Signaling channels allocated: 4
Voice channels allocated: 0
Credits used (rounded-up): 0
Slot: 0/2
Device idx: 0
Dsp Type: DM8147
DSP groups on slot 0/4 slot id 4
dsp 1:
State: UP, firmware: 40.2.11
Max signal/voice channel: 64/64
Max credits: 960, Voice credits: 960, Video credits: 0
num_of_sig_chnls_allocated: 0
Transcoding channels allocated: 12
Group: FLEX_GROUP_VOICE, complexity: FLEX
Shared credits: 507, reserved credits: 0
Signaling channels allocated: 0
Voice channels allocated: 0
Credits used (rounded-up): 0
Group: FLEX_GROUP_XCODE, complexity: HIGH
Shared credits: 0, reserved credits: 180
Transcoding channels allocated: 0
Credits used (rounded-up): 0
Group: FLEX_GROUP_CONF, complexity: CONFERENCE
Shared credits: 0, reserved credits: 183
Codec: CONF_G729, maximum participants: 8
Sessions per dsp: 10
Group: FLEX_GROUP_HW_MTP, complexity: LOW
Shared credits: 0, reserved credits: 90
Transcoding channels allocated: 0
Credits used (rounded-up): 0
Slot: 0/4
Device idx: 0
Dsp Type: SP2700
02-28-2018 05:05 AM
02-28-2018 08:00 AM
Hi There are chances that you might be hitting this bug, one of the DSP is in soft reset state
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd31107/?referring_site=bugquickviewredir
I would say confirm it with TAC.
03-01-2018 12:52 AM
03-01-2018 12:50 AM
Yes. Before suggesting for IOS upgrade, they enabled dsp debugging and captured the logs. But as per the TAC engineer, nothing came of out of it. They gave me some commands to run which actually were not working on 4451 router. Not sure if they even know the 4451 router exists :)
03-01-2018 03:25 AM
03-01-2018 05:21 AM
03-01-2018 08:00 AM
03-06-2018 01:36 AM
Hi Nipun,
I disabled the dsp crashdump as well but no luck. Looks like I would anyway need to upgrade the IOS. I will update once IOS upgrade is done.
Thanks,
Santosh
03-06-2018 03:35 AM
03-10-2018 04:55 AM
I found below command useful just in case if the dsp is stuck in SOFT RESET state.
test dsp device 0/1 1 reset ; where 0/1 is dsp slot number and 1 is dsp id
After executing this command, that particular slot will be reset and the dsp failure will probably recover. I found that after reset; the dsp alarms stopped appearing in the logs
Mar 10 12:35:30.387: %DSPRM-5-UP_DOWN: DSP 1 in subslot 0/1, changed state to down
Mar 10 12:35:32.387: %LINK-5-CHANGED: Interface Service-Engine0/1/0, changed state to administratively down
Mar 10 12:35:38.261: %IOMD-3-MODULE_MESSAGE:iomd: PVDM4-128[0/1] NGIO control packet loss detected: module services resetting
Mar 10 12:35:51.331: dsp_spa_dsp_rp_message_handler: TDL_DSPSPA_MSG_DSP_RESET_COMPLETE.
Mar 10 12:35:51.362: %DSPRM-5-UP_DOWN: DSP 1 in subslot 0/1, changed state to up
Mar 10 12:35:51.372: %DSPRM-5-UP_DOWN: DSP 2 in subslot 0/1, changed state to up
Mar 10 12:35:53.382: %LINK-3-UPDOWN: Interface Service-Engine0/1/0, changed state to up
Mar 10 12:35:54.382: %LINEPROTO-5-UPDOWN: Line protocol on Interface Service-Engine0/1/0, changed state to up
Mar 10 12:36:03.581: %LINEPROTO-5-UPDOWN: Line protocol on Interface Service-Engine0/1/0, changed state to up
INPNVG01#sh voice dsp group all
DSP groups on slot 0/1 slot id 1
dsp 1:
State: UP, firmware: 40.2.11
Max signal/voice channel: 64/64
Max credits: 960, Voice credits: 960, Video credits: 0
num_of_sig_chnls_allocated: 64
Transcoding channels allocated: 0
Group: FLEX_GROUP_VOICE, complexity: FLEX
Shared credits: 960, reserved credits: 0
Signaling channels allocated: 64
Voice channels allocated: 0
Credits used (rounded-up): 0
Slot: 0/1
Device idx: 0
Dsp Type: SP2700
dsp 2:
State: SOFT_RESET, firmware: 40.2.11
Max signal/voice channel: 64/64
Max credits: 960, Voice credits: 960, Video credits: 0
num_of_sig_chnls_allocated: 56
Transcoding channels allocated: 0
Group: FLEX_GROUP_VOICE, complexity: FLEX
Shared credits: 960, reserved credits: 0
Signaling channels allocated: 56
Voice channels allocated: 0
Credits used (rounded-up): 0
Slot: 0/1
Device idx: 0
Dsp Type: SP2700
This is after a successful reset
dsp 2:
State: UP, firmware: 40.2.11
03-10-2018 04:57 AM
03-10-2018 05:02 AM
03-10-2018 06:24 AM
06-10-2018 01:23 PM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide