cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
637
Views
0
Helpful
3
Replies

CUE Crashes During Reload

ilana_ilana
Level 1
Level 1

Hello,

Users are getting fast busy tone when they dial voice mail number.

Its fresh installation, I can ping cue module but not able to access cli interface. On access, its keep showing waiting 1000, 1001, 1002 ....1005.

I tried reload CUE module and its keep reloading with following error

"SAVE TRACE BUFFER
Dec 30 12:37:00 localhost err_handler:   CRASH appsServices startup startup.sh System has crashed. The trace buffer information is stored in the file "atrace_save.log". You can upload the file using "copy log" command "

I found its know BUG CSCup54437 "CUE crashes due to click "Backup /Restore Configuration" GUI but there is no workaround mentioned. Following is complete startup wizard, would appreciate if one can suggest solution:

CME#service-module sm 1/0 reload
Do you want to proceed with reload?[confirm]y
Trying to reload Service Module SM1/0.

CME#
392522: Nov 29 15:13:04.793: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_STDY to SERVICE_MODULE_STATE_SHDN
[Resuming connection 1 to 172.20.11.2 ... ]
INIT: Sending p

shutdown: sending all processes the KILL signal.
shutdown: turning off swap
shutdown: unmounting all file systems
Please stand by md: stopping all md devices.
while rebooting the system.
ACPI: PCI interrupt for device 0000:01:00.0 disabled
Restarting system.




Initializing memory #1. Please wait...




Initializing memory #2. Please wait...


This may take a minute....


 Serial ATA Port 0 : ST949e8k891                            
DDR Memory 4096 MB detected
Intel(R) Core(TM)2 Solo CPU    L3400  @ 1.86GHz
BIOS SM 3.52.8,  BIOS Build date: 02/08/2011
System now booting...

 
Please wait...  

Please press P to select Primary Boot Loader ...   
          or S to select Secondary Boot Loader ...   
          or wait to boot from default configuration ...   

.......................................................................................................
SRE module get located and run....   

Now booting from secondary boot loader....   


Authenticating boot loader....   


Secondary Boot Loader authenticated - booting....   



Please enter '***' to change boot configuration:
***

 ServicesEngine Bootloader Version : 2.1.36


ServicesEngine boot-loader> en                                                 
Unrecognized command

ServicesEngine boot-loader> boot                                               
Incomplete command
Usage: boot <disk | helper | diag | chainloader>

ServicesEngine boot-loader> boot disk                                          
Loading disk:/bzImage ... Verifying ... done.
Starting Kernel.

Platform: sm
Verifying application level programs
Application level programs verification OK!
INIT: version 2.86 booting
mounting proc fs ...
mounting sys fs ...
mounting /dev/shm tmpfs ...
reiser root fs ...
Reiserfs super block in block 16 on 0x801 of format 3.6 with standard journal
Blocks (total/free): 122096000/121981503 by 4096 bytes
Filesystem is clean
Filesystem seems mounted read-only. Skipping journal replay.
sd 4:0:0:0: [sdb] Assuming drive cache: write through
sd 4:0:0:0: [sdb] Assuming drive cache: write through

Checking internal tree..finished

FILESYSTEM CLEAN
Remounting the root filesystem read-write...

kernel.sem = 1900 4000 32 100
vm.overcommit_memory = 1
vm.min_free_kbytes = 8192


                Welcome to Cisco Service Engine

Setting the system time from hardware clock

********** rc.aesop ****************
Populating resource values from /etc/sm_rsrc_file
Populating resource values from /etc/default_rsrc_file
Populating resource values from /etc/products/cue/default_rsrc_file
Populating resource values from /etc/products/cue/sm_rsrc_file
Push button monitor started
Processing manifests . . . . . . . . . . . . . complete
==> Management interface is eth0
==> Management interface is eth0

392523: Nov 29 15:16:39.951: %LINEPROTO-5-UPDOWN: Line protocol on Interface SM1/1, changed state to down
392524: Nov 29 15:16:39.951: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
392525: Nov 29 15:16:41.283: %LINEPROTO-5-UPDOWN: Line protocol on Interface SM1/1, changed state to up
392526: Nov 29 15:16:42.867: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_SHDN to SERVICE_MODULE_STATE_STDY


Serial Number: F0B1829873
Disk /dev/sdb doesn't contain a valid partition table
INIT: Entering runlevel: 2
********** rc.post_install ****************
INIT: Switching to runlevel: 4
INIT: Sending processes the TERM signal
==> Starting CDP
STARTED: ntp_startup.sh
STARTED: LDAP_startup.sh
STARTED: SQL_startup.sh
STARTED: dwnldr_startup.sh
STARTED: HTTP_startup.sh
STARTED: probe
STARTED: fndn_udins_wrapper
STARTED: superthread_startup.sh
STARTED: /usr/wfavvid/run-wfengine.sh
STARTED: /usr/bin/launch_ums.sh

 Waiting 17 ...
 Waiting 203 ...5:17:11.284: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
 Waiting 273 ...
 Waiting 797 ...
 Waiting 2001 ...syslog-ng/phase                         100
syslog-ng/alive                         1
platform_config/phase                   100
platform_config/alive                   1
trace/phase                             100
trace/alive                             1
rbcp/phase                              100
rbcp/alive                              1
ntp/phase                               21
ntp/alive                               1
cdp/phase                               100
cdp/alive                               1
ldap/phase                              100
ldap/alive                              1
sql/phase                               20
sql/alive                               1
downloader/phase                        100
downloader/alive                        1
http/phase                              25
http/alive                              1
probe/phase                             100
probe/alive                             1
udins/phase                             100
udins/alive                             1
mgmt/phase                              41
mgmt/alive                              1
snmp/phase                              41
snmp/alive                              1
superthread/phase                       100
superthread/alive                       1
dns/phase                               100
dns/alive                               1
backuprestore/phase                     100
backuprestore/alive                     1
usermanager/phase                       22
usermanager/alive                       1
dbclient/phase                          22
dbclient/alive                          1
ccn_config/phase                        25
ccn_config/alive                        1
smtp/phase                              29
smtp/alive                              1
llama/phase                             23
llama/alive                             1
ccn/phase                               41
ccn/alive                               1
ums/phase                               22
ums/alive                               1
cli/phase                               28
cli/alive                               1
usermanager2/phase                      25
usermanager2/alive                      1
entitymanager/phase                     25
entitymanager/alive                     1
configapi/phase                         29
configapi/alive                         1
gen-scheduler/phase                     100
gen-scheduler/alive                     1
webapp/phase                            29
webapp/alive                            1
umg-registration/phase                  29
umg-registration/alive                  1
umg-directory/phase                     29
umg-directory/alive                     1
sitemanager/phase                       29
sitemanager/alive                       1
TIMEOUT
It looks like things are stuck, trying a monitor interrupt.
MONITOR EXITING...
SAVE TRACE BUFFER
Dec 30 12:37:00 localhost err_handler:   CRASH appsServices startup startup.sh System has crashed. The trace buffer information is stored in the file "atrace_save.log". You can upload the file using "copy log" command
INIT: Sending processes the TERM signal

Sending an RBCP message to IOS notifying module reboot...

Rebooting ...

shutdown: sending all processes the TERM signal...
rbcp:    INFO rbcp daemon output END

trace:    INFO trace daemon output END

platform.config:    INFO platform.config server output END


392528: Nov 29 15:50:39.298: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_STDY to SERVICE_MODULE_STATE_SHDN
392529: Nov 29 15:50:39.298: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_STDY to SERVICE_MODULE_STATE_SHDNshutdown: sending all processes the KILL signal.
shutdown: turning off swap
shutdown: unmounting all file systems
Please stand by md: stopping all md devices.
while rebooting the system.
ACPI: PCI interrupt for device 0000:01:00.0 disabled
Restarting system.




Initializing memory #1. Please wait...




Initializing memory #2. Please wait...


This may take a minute....


 Serial ATA Port 0 : ST9500620NS                             
DDR Memory 4096 MB detected
Intel(R) Core(TM)2 Solo CPU    L3400  @ 1.86GHz
BIOS SM 3.52.8,  BIOS Build date: 02/08/2011
System now booting...

 
Please wait...  

Please press P to select Primary Boot Loader ...   
          or S to select Secondary Boot Loader ...   
          or wait to boot from default configuration ...   

.......................................................................................................
SRE module get located and run....   

Now booting from secondary boot loader....   


Authenticating boot loader....   


Secondary Boot Loader authenticated - booting....   



Please enter '***' to change boot configuration:

Detect and Initialize network device


Backup current platform configurations....

SRE step 1 - SM registration...
Finding (hd1,3)/296e03bc-3236-4a68-a178-688e56400a1e, failed
Local install not supported

Response - no installation needed (len: 422)

SRE Installation Not Needed


Restoring orignial configuration...

Updating flash with bootloader configuration.
Please wait ................... done.

392530: Nov 29 15:52:09.819: %SM_INSTALL-6-INST_RBIP: SM1/0 received msg: RBIP Registration RequestLoading disk:/bzImage ... Verifying ... done.
Starting Kernel.

Platform: sm
Verifying application level programs
Application level programs verification OK!
INIT: version 2.86 booting
mounting proc fs ...
mounting sys fs ...
mounting /dev/shm tmpfs ...
reiser root fs ...
Reiserfs super block in block 16 on 0x801 of format 3.6 with standard journal
Blocks (total/free): 122096000/121981137 by 4096 bytes
Filesystem is clean
Filesystem seems mounted read-only. Skipping journal replay.
sd 4:0:0:0: [sdb] Assuming drive cache: write through
sd 4:0:0:0: [sdb] Assuming drive cache: write through
Checking internal tree..finished

FILESYSTEM CLEAN
Remounting the root filesystem read-write...

kernel.sem = 1900 4000 32 100
vm.overcommit_memory = 1
vm.min_free_kbytes = 8192


                Welcome to Cisco Service Engine

Setting the system time from hardware clock

********** rc.aesop ****************
Populating resource values from /etc/sm_rsrc_file
Populating resource values from /etc/default_rsrc_file
Populating resource values from /etc/products/cue/default_rsrc_file
Populating resource values from /etc/products/cue/sm_rsrc_file
Push button monitor started
Processing manifests . . . . . . . . . . . . . complete
==> Management interface is eth0
==> Management interface is eth0

392531: Nov 29 15:53:15.168: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_MODULE_STATE_SHDN to SERVICE_MODULE_STATE_STDY


Serial Number: FOC18376WEH
Disk /dev/sdb doesn't contain a valid partition table
INIT: Entering runlevel: 2
********** rc.post_install ****************
INIT: Switching to runlevel: 4
INIT: Sending processes the TERM signal
==> Starting CDP
STARTED: ntp_startup.sh
STARTED: LDAP_startup.sh
STARTED: SQL_startup.sh
STARTED: dwnldr_startup.sh
STARTED: HTTP_startup.sh
STARTED: probe
STARTED: fndn_udins_wrapper
STARTED: superthread_startup.sh
STARTED: /usr/wfavvid/run-wfengine.sh
STARTED: /usr/bin/launch_ums.sh

 Waiting 35 ...
 Waiting 413 ...
 Waiting 959 ...

 Waiting 1004 ...

 Waiting 1005 ...


3 Replies 3

Manish Gogna
Cisco Employee
Cisco Employee

Hi Ilana,

The bug  CSCup54437   is applicable for scenarios where backup is initiated via GUI and it causes the crash, the workaround for this bug is to initiate backup via CLI. In your case the trigger is different so its probably a different issue. One of the imp errors that i see is the following

Dec 30 12:37:00 localhost err_handler:   CRASH appsServices startup startup.sh System has crashed. The trace buffer information is stored in the file "atrace_save.log". You can upload the file using "copy log" command
INIT: Sending processes the TERM signal

I have see some cases like this requiring a CUE reinstall, however i would suggest checking with TAC for their recommendation if there are no additional inputs on this post.

Manish

Thank Manish,

I have opened TAC will update with result.

Regards,

Hi ilana.

Did you find the cause for this problem and his possible fix ?

Regards.