ACE appliance Crash question

Joe Dunn · ‎12-28-2011

Does anyone have an idea what the cause of these error messages would be? I searched CCO to see if I could locate what they are and mean (technical definition). I was only able to locate that they could be an output (you don't say).

Aug 31 2011 18:28:26 : %ACE-2-727012: HA: FT Group 1 changed state to FSM_FT_STATE_STANDBY_REAP. Event: FSM_FT_EV_RELINQUISH

Aug 31 2011 18:28:48 : %ACE-2-727012: HA: FT Group 1 changed state to FSM_FT_STATE_STANDBY_BULK. Event: FSM_FT_EV_REAP_DONE

Aug 31 2011 18:28:50 : %ACE-2-727012: HA: FT Group 2 changed state to FSM_FT_STATE_STANDBY_REAP. Event: FSM_FT_EV_RELINQUISH

Dec 25 2011 15:40:44 : %ACE-2-727012: HA: FT Group 2 changed state to FSM_FT_STATE_ACTIVE. Event: FSM_FT_EV_PEER_DOWN

Dec 26 2011 10:25:21 : %ACE-2-727013: HA: Peer 1 is UP and reachable.

I had an ACE appliance lock and cause a failover. The primary device's light was flashing orange. The console was unresponsive. I had to power cycle it to restore access. This also cleared whatever the issue was. Since it was power cycled, I lost the syslog. I checked the core and 2 other folders looking for any crashinfo file. None were created. Is there anyway to figure out what could have caused this to lock up?

Borys Berlog · ‎01-03-2012

Hi Joe

These are actually informational messages

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_1_0/configuration/system/message/guide/messags.html#wp1144125

My guess you have tracking configured on ACE, when ACE#1 got stuck it looses tracking (but haven't got completely stuck and still able to comunicate to peer) and inititated a switchower (Event FSM_FT_EV_RELINQUISH)

then it got compeltely dead and you see event : FSM_FT_EV_PEER_DOWN

This is basically all what you can get from these syslog messageses.

If you don't see any files on disk and core: directory, it makes almost impossible to figure out what was a problem cause.

Can you please show sh ver output ?

Joe Dunn · ‎01-03-2012

Here's the show ver. I suspect you may be looking for an "undocumented feature" perhaps? :?)

sh ver
Cisco Application Control Software (ACSW)
TAC support: http://www.cisco.com/tac
Copyright (c) 1985-2011 by Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
loader: Version 0.95.1
system: Version A4(2.1a) [build 3.0(0)A4(2.1a) adbuild_21:41:15-2011/07/21_
/auto/adbure_nightly4/renumber/rel_a4_2_1_throttle/REL_3_0_0_A4_2_1A]
system image file: (hd0,1)/c4710ace-t1k9-mz.A4_2_1a.bin
Device Manager version 4.2 (0) 20110629:0926

installed license: no feature license is installed

Hardware
cpu info:
    Motherboard:
        number of cpu(s): 2
    Daughtercard:
        number of cpu(s): 16
memory info:
    total: 6226372 kB, free: 4267036 kB
    shared: 0 kB, buffers: 22724 kB, cached 0 kB
cf info:
    filesystem: /dev/hdb2
    total: 861668 kB, used: 787580 kB, available: 30316 kB

last boot reason: Unknown
configuration register: 0x1
whrlbpraln01 kernel uptime is 7 days 21 hours 43 minute(s) 50 second(s)

whrlbpraln01/Admin#

Joe Dunn · ‎01-03-2012

Thank you for your response.Here's the show ver. I suspect you may be looking for an undocumented feature perhaps?

sh ver
Cisco Application Control Software (ACSW)
TAC support:

Copyright (c) 1985-2011 by Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at

Software
loader: Version 0.95.1
system: Version A4(2.1a) [build 3.0(0)A4(2.1a) adbuild_21:41:15-2011/07/21_
/auto/adbure_nightly4/renumber/rel_a4_2_1_throttle/REL_3_0_0_A4_2_1A]
system image file: (hd0,1)/c4710ace-t1k9-mz.A4_2_1a.bin
Device Manager version 4.2 (0) 20110629:0926

installed license: no feature license is installed

Hardware
cpu info:
    Motherboard:
        number of cpu(s): 2
    Daughtercard:
        number of cpu(s): 16
memory info:
    total: 6226372 kB, free: 4267036 kB
    shared: 0 kB, buffers: 22724 kB, cached 0 kB
cf info:
    filesystem: /dev/hdb2
    total: 861668 kB, used: 787580 kB, available: 30316 kB

last boot reason: Unknown
configuration register: 0x1
kernel uptime is 7 days 21 hours 43 minute(s) 50 second(s)

Jorge Bejarano · ‎01-03-2012

Hello Joe,

It is hard to say why you got that crash, if the device did not generate any core dump file and the last reboot reason is unknown.

There are some "silent" bugs which show last reboot reason: unknown but we cannot say that's for sure in your case.

You can proactively proceed with the software upgrade, here you have the link where you can get the newest version:

http://www.cisco.com/cisco/software/release.html?mdfid=281222179&catid=268437639&softwareid=282775307&release=A5%281.1%29&rellifecycle=&relind=AVAILABLE&reltype=latest

Additionally, you can monitor the behavior and the issue re-occurs and if it does not generate any core dumps or a different last reboot reason, then you can proceed with the replacement of the device to discard any hardware failure.

Borys Berlog · ‎01-04-2012

Joe Dunn wrote:

Thank you for your response.Here's the show ver. I suspect you may be looking for an undocumented feature perhaps?

Hi Joe, no I was looking for SW version and reload reason. And unfortunately I have nothing to add to Jorge comment.

Joe Dunn · ‎01-04-2012

I thank you guys for taking a quick look to see if you're able to provide any answers. The ACE appliance that locked up has been running perfectly since the reload. I'll chalk it up to solar flares

Jorge Bejarano · ‎01-04-2012

Sure and have a happy new year!!!