do i have a dead ACE module?

[michaellambert1980]

I rebooted one of my HA ACE modules and it hasn't come back up.  The logs on the 6500 show the following..

Mar 23 08:54:25: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 08:54:28: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 08:54:28: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 08:54:27: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 08:54:28: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 08:54:43: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 08:55:18: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 08:57:30: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:07:23: %SNMP-5-MODULETRAP: Module 4 [Down] Trap
Mar 23 09:07:23: SP: The PC in slot 4 is shutting down. Please wait ...
Mar 23 09:07:56: SP: PC shutdown completed for module 4
Mar 23 09:08:06: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set off (Fabric channel errors)
Mar 23 09:15:48: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 09:15:50: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 09:15:51: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 09:15:51: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 09:15:51: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 09:16:06: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:16:41: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 09:17:45: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:28:00: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:28:46: %SNMP-5-MODULETRAP: Module 4 [Down] Trap
Mar 23 09:28:46: SP: The PC in slot 4 is shutting down. Please wait ...
Mar 23 09:29:19: SP: PC shutdown completed for module 4
Mar 23 09:29:29: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set off (Fabric channel errors)
Mar 23 09:37:11: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 09:37:13: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 09:37:13: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 09:37:12: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 09:37:13: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 09:37:28: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:38:03: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 09:38:15: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1

The output of the ACE console is the following....

System Bootstrap, Version 12.2[123],
Copyright (c) 1994-2009 by cisco Systems, Inc.
Slot 4 : Running DEFAULT rommon image ...

.ACE platform with 1048576 Kbytes of main memory

.Loading disk0:c6ace-t1k9-mz.A2_3_4.bin.  Please wait ....
Uncompressing Linux...
Starting the kernel...
INIT: version 2.78 booting
Mounting Second Ramdisk ....
Second Ramdisk successfully mounted
Configuring network interfaces.
CF dump: Register callback functions
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
/dev/cf: 11 files, 26575/63414 clusters
FAT FS is ok
Compact Flash size 1014624(in 1k blocks) ...
Core file size 204800
Available free size in cf is 589424 (in 1k blocks) ...
set_coredump 2.11, 12 Mar 2005, FAT32, LFN
first_cluster = 0x5102 num_cluster = 0x40 (64)
inserting procfs
inserting isan_kthread
inserting wiremod
inserting klib
inserting resdrv
inserting tlv
inserting sse
inserting kpss
inserting sdwrap
creating sdwrap device
inserting klm_tl
creating tl device
inserting klm_scp
inserting klm_mts
creating mts0 device
creating mtscfg0 device
inserting utaker
creating utaker0 device
creating utaker1 device
inserting sysmgr-hb
creating sysmgr-hb device
inserting modlock
creating modlock device
inserting bufmgr
inserting pkt_fifo
inserting encdec
creating encdec device
inserting pseudo
inserting drammap mod
creating drammap device
inserting ixp_dnld
creating ixp_dnld device
inserting sysdrv
creating sysdrv device
New registry installed.
INIT: Entering runlevel: 3
inserting i2c module
inserting ssa driver
inserting cde driver
inserting bf_dnld driver
inserting pfm_drv driver
inserting regaccess driver
inserting bf_nvram driver

Firmware compiled 21-Jan-11 13:14 by integ Build [25600]

ACE Daughter boards DB1 not present DB2 not present.
downloading fpga to cde 1

Read 3262454 bytes from ./cde1_core.bit
FPGA Date: 2007/12/18 Time: 14:22: 0

CDE 1 download successful
downloading fpga to cde 2

Read 2377744 bytes from ./cde2_core.bit
FPGA Date: 2007/ 8/15 Time: 20:59:47

CDE 2 download successful
FPGA Programming Done

CDE 1 revision ID 0403
CDE 2 revision ID 0402
enabling cde 0 interrupts
finished CDE setup

Configuring NP 1 Memory
Configuring NP 2 Memory
.............................

Waiting for NP 1 SRAM memory to clear...success

Downloading NP 1 Image
.............................

Waiting for NP 2 SRAM memory to clear...success

Downloading NP 2 Image
..... 0x4eef60 (5173088) bytes downloaded

..... 0x4eef60 (5173088) bytes downloaded

Loading Nitrox driver.
PCI device 177d:0002
Writing register at address 3838 with e00
size = 8108
Ctx memory range(0x0000000-0x10000000)
Cleared 262144 1024-byte blocks in 5 requests.
N2SetupMicrocode: failed; error code 3
Writing register at address 3898 with 1
N2LoadMicrocode: failed; error code 3

N2LoadMicrocode: failed; error code 3

[sivaksiv · Accepted solution]

Hi,

Looks like its a hardware problem. Try reseating the module and see if that helps. If it still doesn't boot have a TAC case raised for further analysis and requires a replacement.

-

Siva

[michaellambert1980]

Thanks Siva.  Have tried the reseat but still no luck.  TAC case in progress.  I think I've had some back luck with these as this is the second module to have gone faulty in as many weeks. 

[Akhtar Samo]

Hi Mick,

Any update from TAC on this ? I have also received following logs on the switch and the active ace had recovered by itself and is now working fine.

%SVCLC-5-SVCLCNTP: Could not update clock on the module 11, rc is -1

Strange thing is the system and kernel uptimes are different ?

# show system up time

System start time:          Tue Jun 12 10:41:12 2012

System uptime:              1 days, 2 hours, 24 minutes, 33 seconds

Kernel uptime:              5 days, 7 hours, 25 minutes, 33 seconds

There are no messages on the switch indicating a module reload but it has caused an outage.

Regards,

Akhtar

[Jorge Bejarano]

Hello Akhtar,

Can you upload the command: #show version?

Can you upload the dir core: , hopefully the ACE might have generated some core dumps which might help us to determine the failure?

Here you have a link about getting the core dumps:

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Overview_of_ACE_Troubleshooting#Copying_Core_Dumps

Have you experienced this issue before? Did you experience this issue during a high peak of traffic?

Did you apply any change in the configuration?

#show tech-support and core dumps would help to determine if this was a hardware failure or a software defect

-----------------------------------------------------------

Jorge

[nicataliyev]

I've the same problem. if your module has been produced between 2005 - 2010 then Cisco has field notice about that. HW on referred list has memory component problem. It should be fixed asap.

http://www.cisco.com/c/en/us/support/docs/field-notices/637/fn63782.html

[expertadvisor20151]

Cisco ITD might be a good fit here:

http://blogs.cisco.com/datacenter/itd-load-balancing-traffic-steering-clustering-using-nexus-5k6k7k