11-29-2015 10:39 AM - edited 03-17-2019 05:03 AM
Hello,
Users are getting fast busy tone when they dial voice mail number.
Its fresh installation, I can ping cue module but not able to access cli interface. On access, its keep showing waiting 1000, 1001, 1002 ....1005.
I tried reload CUE module and its keep reloading with following error
"SAVE TRACE BUFFER
Dec 30 12:37:00 localhost err_handler: CRASH appsServices startup startup.sh System has crashed. The trace buffer information is stored in the file "atrace_save.log". You can upload the file using "copy log" command "
I found its know BUG CSCup54437 "CUE crashes due to click "Backup /Restore Configuration" GUI but there is no workaround mentioned. Following is complete startup wizard, would appreciate if one can suggest solution:
CME#service-module sm 1/0 reload
Do you want to proceed with reload?[confirm]y
Trying to reload Service Module SM1/0.
CME#
392522: Nov 29 15:13:04.793: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_STDY to SERVICE_MODULE_STATE_SHDN
[Resuming connection 1 to 172.20.11.2 ... ]
INIT: Sending p
shutdown: sending all processes the KILL signal.
shutdown: turning off swap
shutdown: unmounting all file systems
Please stand by md: stopping all md devices.
while rebooting the system.
ACPI: PCI interrupt for device 0000:01:00.0 disabled
Restarting system.
Initializing memory #1. Please wait...
Initializing memory #2. Please wait...
This may take a minute....
Serial ATA Port 0 : ST949e8k891
DDR Memory 4096 MB detected
Intel(R) Core(TM)2 Solo CPU L3400 @ 1.86GHz
BIOS SM 3.52.8, BIOS Build date: 02/08/2011
System now booting...
Please wait...
Please press P to select Primary Boot Loader ...
or S to select Secondary Boot Loader ...
or wait to boot from default configuration ...
.......................................................................................................
SRE module get located and run....
Now booting from secondary boot loader....
Authenticating boot loader....
Secondary Boot Loader authenticated - booting....
Please enter '***' to change boot configuration:
***
ServicesEngine Bootloader Version : 2.1.36
ServicesEngine boot-loader> en
Unrecognized command
ServicesEngine boot-loader> boot
Incomplete command
Usage: boot <disk | helper | diag | chainloader>
ServicesEngine boot-loader> boot disk
Loading disk:/bzImage ... Verifying ... done.
Starting Kernel.
Platform: sm
Verifying application level programs
Application level programs verification OK!
INIT: version 2.86 booting
mounting proc fs ...
mounting sys fs ...
mounting /dev/shm tmpfs ...
reiser root fs ...
Reiserfs super block in block 16 on 0x801 of format 3.6 with standard journal
Blocks (total/free): 122096000/121981503 by 4096 bytes
Filesystem is clean
Filesystem seems mounted read-only. Skipping journal replay.
sd 4:0:0:0: [sdb] Assuming drive cache: write through
sd 4:0:0:0: [sdb] Assuming drive cache: write through
Checking internal tree..finished
FILESYSTEM CLEAN
Remounting the root filesystem read-write...
kernel.sem = 1900 4000 32 100
vm.overcommit_memory = 1
vm.min_free_kbytes = 8192
Welcome to Cisco Service Engine
Setting the system time from hardware clock
********** rc.aesop ****************
Populating resource values from /etc/sm_rsrc_file
Populating resource values from /etc/default_rsrc_file
Populating resource values from /etc/products/cue/default_rsrc_file
Populating resource values from /etc/products/cue/sm_rsrc_file
Push button monitor started
Processing manifests . . . . . . . . . . . . . complete
==> Management interface is eth0
==> Management interface is eth0
392523: Nov 29 15:16:39.951: %LINEPROTO-5-UPDOWN: Line protocol on Interface SM1/1, changed state to down
392524: Nov 29 15:16:39.951: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
392525: Nov 29 15:16:41.283: %LINEPROTO-5-UPDOWN: Line protocol on Interface SM1/1, changed state to up
392526: Nov 29 15:16:42.867: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_SHDN to SERVICE_MODULE_STATE_STDY
Serial Number: F0B1829873
Disk /dev/sdb doesn't contain a valid partition table
INIT: Entering runlevel: 2
********** rc.post_install ****************
INIT: Switching to runlevel: 4
INIT: Sending processes the TERM signal
==> Starting CDP
STARTED: ntp_startup.sh
STARTED: LDAP_startup.sh
STARTED: SQL_startup.sh
STARTED: dwnldr_startup.sh
STARTED: HTTP_startup.sh
STARTED: probe
STARTED: fndn_udins_wrapper
STARTED: superthread_startup.sh
STARTED: /usr/wfavvid/run-wfengine.sh
STARTED: /usr/bin/launch_ums.sh
Waiting 17 ...
Waiting 203 ...5:17:11.284: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
Waiting 273 ...
Waiting 797 ...
Waiting 2001 ...syslog-ng/phase 100
syslog-ng/alive 1
platform_config/phase 100
platform_config/alive 1
trace/phase 100
trace/alive 1
rbcp/phase 100
rbcp/alive 1
ntp/phase 21
ntp/alive 1
cdp/phase 100
cdp/alive 1
ldap/phase 100
ldap/alive 1
sql/phase 20
sql/alive 1
downloader/phase 100
downloader/alive 1
http/phase 25
http/alive 1
probe/phase 100
probe/alive 1
udins/phase 100
udins/alive 1
mgmt/phase 41
mgmt/alive 1
snmp/phase 41
snmp/alive 1
superthread/phase 100
superthread/alive 1
dns/phase 100
dns/alive 1
backuprestore/phase 100
backuprestore/alive 1
usermanager/phase 22
usermanager/alive 1
dbclient/phase 22
dbclient/alive 1
ccn_config/phase 25
ccn_config/alive 1
smtp/phase 29
smtp/alive 1
llama/phase 23
llama/alive 1
ccn/phase 41
ccn/alive 1
ums/phase 22
ums/alive 1
cli/phase 28
cli/alive 1
usermanager2/phase 25
usermanager2/alive 1
entitymanager/phase 25
entitymanager/alive 1
configapi/phase 29
configapi/alive 1
gen-scheduler/phase 100
gen-scheduler/alive 1
webapp/phase 29
webapp/alive 1
umg-registration/phase 29
umg-registration/alive 1
umg-directory/phase 29
umg-directory/alive 1
sitemanager/phase 29
sitemanager/alive 1
TIMEOUT
It looks like things are stuck, trying a monitor interrupt.
MONITOR EXITING...
SAVE TRACE BUFFER
Dec 30 12:37:00 localhost err_handler: CRASH appsServices startup startup.sh System has crashed. The trace buffer information is stored in the file "atrace_save.log". You can upload the file using "copy log" command
INIT: Sending processes the TERM signal
Sending an RBCP message to IOS notifying module reboot...
Rebooting ...
shutdown: sending all processes the TERM signal...
rbcp: INFO rbcp daemon output END
trace: INFO trace daemon output END
platform.config: INFO platform.config server output END
392528: Nov 29 15:50:39.298: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_STDY to SERVICE_MODULE_STATE_SHDN
392529: Nov 29 15:50:39.298: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_STDY to SERVICE_MODULE_STATE_SHDNshutdown: sending all processes the KILL signal.
shutdown: turning off swap
shutdown: unmounting all file systems
Please stand by md: stopping all md devices.
while rebooting the system.
ACPI: PCI interrupt for device 0000:01:00.0 disabled
Restarting system.
Initializing memory #1. Please wait...
Initializing memory #2. Please wait...
This may take a minute....
Serial ATA Port 0 : ST9500620NS
DDR Memory 4096 MB detected
Intel(R) Core(TM)2 Solo CPU L3400 @ 1.86GHz
BIOS SM 3.52.8, BIOS Build date: 02/08/2011
System now booting...
Please wait...
Please press P to select Primary Boot Loader ...
or S to select Secondary Boot Loader ...
or wait to boot from default configuration ...
.......................................................................................................
SRE module get located and run....
Now booting from secondary boot loader....
Authenticating boot loader....
Secondary Boot Loader authenticated - booting....
Please enter '***' to change boot configuration:
Detect and Initialize network device
Backup current platform configurations....
SRE step 1 - SM registration...
Finding (hd1,3)/296e03bc-3236-4a68-a178-688e56400a1e, failed
Local install not supported
Response - no installation needed (len: 422)
SRE Installation Not Needed
Restoring orignial configuration...
Updating flash with bootloader configuration.
Please wait ................... done.
392530: Nov 29 15:52:09.819: %SM_INSTALL-6-INST_RBIP: SM1/0 received msg: RBIP Registration RequestLoading disk:/bzImage ... Verifying ... done.
Starting Kernel.
Platform: sm
Verifying application level programs
Application level programs verification OK!
INIT: version 2.86 booting
mounting proc fs ...
mounting sys fs ...
mounting /dev/shm tmpfs ...
reiser root fs ...
Reiserfs super block in block 16 on 0x801 of format 3.6 with standard journal
Blocks (total/free): 122096000/121981137 by 4096 bytes
Filesystem is clean
Filesystem seems mounted read-only. Skipping journal replay.
sd 4:0:0:0: [sdb] Assuming drive cache: write through
sd 4:0:0:0: [sdb] Assuming drive cache: write through
Checking internal tree..finished
FILESYSTEM CLEAN
Remounting the root filesystem read-write...
kernel.sem = 1900 4000 32 100
vm.overcommit_memory = 1
vm.min_free_kbytes = 8192
Welcome to Cisco Service Engine
Setting the system time from hardware clock
********** rc.aesop ****************
Populating resource values from /etc/sm_rsrc_file
Populating resource values from /etc/default_rsrc_file
Populating resource values from /etc/products/cue/default_rsrc_file
Populating resource values from /etc/products/cue/sm_rsrc_file
Push button monitor started
Processing manifests . . . . . . . . . . . . . complete
==> Management interface is eth0
==> Management interface is eth0
392531: Nov 29 15:53:15.168: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_SHDN to SERVICE_MODULE_STATE_STDY
Serial Number: FOC18376WEH
Disk /dev/sdb doesn't contain a valid partition table
INIT: Entering runlevel: 2
********** rc.post_install ****************
INIT: Switching to runlevel: 4
INIT: Sending processes the TERM signal
==> Starting CDP
STARTED: ntp_startup.sh
STARTED: LDAP_startup.sh
STARTED: SQL_startup.sh
STARTED: dwnldr_startup.sh
STARTED: HTTP_startup.sh
STARTED: probe
STARTED: fndn_udins_wrapper
STARTED: superthread_startup.sh
STARTED: /usr/wfavvid/run-wfengine.sh
STARTED: /usr/bin/launch_ums.sh
Waiting 35 ...
Waiting 413 ...
Waiting 959 ...
Waiting 1004 ...
Waiting 1005 ...
11-29-2015 07:14 PM
Hi Ilana,
The bug CSCup54437 is applicable for scenarios where backup is initiated via GUI and it causes the crash, the workaround for this bug is to initiate backup via CLI. In your case the trigger is different so its probably a different issue. One of the imp errors that i see is the following
Dec 30 12:37:00 localhost err_handler: CRASH appsServices startup startup.sh System has crashed. The trace buffer information is stored in the file "atrace_save.log". You can upload the file using "copy log" command
INIT: Sending processes the TERM signal
I have see some cases like this requiring a CUE reinstall, however i would suggest checking with TAC for their recommendation if there are no additional inputs on this post.
Manish
11-29-2015 11:17 PM
Thank Manish,
I have opened TAC will update with result.
Regards,
02-11-2016 01:23 PM
Hi ilana.
Did you find the cause for this problem and his possible fix ?
Regards.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide