ACE10 Strange behaviour

[Akhtar Samo]

Hello,

Its strange to see the big difference in the system uptime and the kernel uptime. The ACE had caused a production impact for around 8 minutes and the standby ace didn't took over during that time frame although the FT/query vlan is configured perfectly fine.

Since there was no log generated on the 6500 switch for the module reset, i suspect that the module would have got hung and recovered by itself.

I also dont find any thing strange in the ft history * outputs.

I suspect that this might be a bug since the image is very old (Version A2(1.0))

`show system uptime`

System start time:          Tue Jun 12 10:41:12 2012

System uptime:              0 days, 20 hours, 5 minutes, 6 seconds

Kernel uptime:              5 days, 1 hours, 6 minutes, 8 seconds

last boot reason:  Unknown

configuration register:  0x1

ACE-1 kernel uptime is 5 days 1 hours 6 minute(s) 8 second(s)

`show ft peer detail`

Peer Id                      : 1

State                        : FSM_PEER_STATE_COMPATIBLE

Maintenance mode             : MAINT_MODE_OFF

FT Vlan                      : 503

FT Vlan IF State             : UP

My IP Addr                   : 2.2.2.1

Peer IP Addr                 : 2.2.2.2

Query Vlan                   : 502

Query Vlan IF State          : UP

Peer Query IP Addr           : 5.5.5.2

Heartbeat Interval           : 200

Heartbeat Count              : 20

Tx Packets                   : 14870

Tx Bytes                     : 3459966

Rx Packets                   : 14674

Rx Bytes                     : 3443749

Rx Error Bytes               : 0

Tx Keepalive Packets         : 14520

Rx Keepalive Packets         : 14520

TL_CLOSE count               : 0

FT_VLAN_DOWN count           : 0

PEER_DOWN count              : 0

SRG Compatibility            : COMPATIBLE

License Compatibility        : COMPATIBLE

FT Groups                    : 9

`show ft group detail`

FT Group                     : 1

No. of Contexts              : 1

Context Name                 : Admin

Context Id                   : 0

Configured Status            : in-service

Maintenance mode             : MAINT_MODE_OFF

My State                     : FSM_FT_STATE_ACTIVE

My Config Priority           : 250

My Net Priority              : 250

My Preempt                   : Enabled

Peer State                   : FSM_FT_STATE_STANDBY_HOT

Peer Config Priority         : 100

Peer Net Priority            : 100

Peer Preempt                 : Enabled

Peer Id                      : 1

Last State Change time       : Tue Jun 12 10:43:20 2012

Running cfg sync enabled     : Enabled

Running cfg sync status      : Running configuration sync has completed

Startup cfg sync enabled     : Enabled

Startup cfg sync status      : Startup configuration sync has completed

Bulk sync done for ARP: 0

Bulk sync done for LB: 0

Bulk sync done for ICM: 0

FT Group                     : 2

No. of Contexts              : 1

Context Name                 : Microsoft

Context Id                   : 2

Configured Status            : in-service

Maintenance mode             : MAINT_MODE_OFF

My State                     : FSM_FT_STATE_ACTIVE

My Config Priority           : 250

My Net Priority              : 250

My Preempt                   : Enabled

Peer State                   : FSM_FT_STATE_STANDBY_HOT

Peer Config Priority         : 100

Peer Net Priority            : 100

Peer Preempt                 : Enabled

Peer Id                      : 1

Last State Change time       : Tue Jun 12 10:43:20 2012

Running cfg sync enabled     : Enabled

Running cfg sync status      : Running configuration sync has completed

Startup cfg sync enabled     : Enabled

Startup cfg sync status      : Startup configuration sync has completed

Bulk sync done for ARP: 0

Bulk sync done for LB: 0

Bulk sync done for ICM: 0

Switch logs:

%SVCLC-5-SVCLCNTP: Could not update clock on the module 11, rc is -1

Regards,

Akhtar

[Jorge Bejarano]

Hello Akhtar,

As you said, probably the device might have started hunging at that moment then that´s why the failover was never fired, it would have been good to force a manual reset of the module.

There are some bugs which show: "last reboot reason: unknown" and they are called: "silent bugs" however the ACE might have had a process which was stuck at that moment. Do you have a high logging level?

Also you can check with: # dir core: to see if the device generated any core dump, here you have the link about it:

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Overview_of_ACE_Troubleshooting#Copying_Core_Dumps

Anyway, if the device did not generate any core dump, it will be good if you proceed with a proactive upgrade to the version:a2.3.3 or higher and monitor the behavior, in case you experience the same behavior, please try to collect #show tech-support if it is possible, if not hopefully the ACE will failover to its peer but it does not happen, force the reboot and trigger the failover and avoid further outage, but please be aware that as much information we got it will be better to determine the root cause.

Here you have the link where you can get the software from:

http://www.cisco.com/cisco/software/release.html?mdfid=280557289&softwareid=280836740&release=A2%283.6a%29&flowid=3314

Here you have a link about the upgrade process:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A1/configuration/administration/guide/upgrade.html#wp1008104

Jorge