03-23-2012 02:42 AM
I rebooted one of my HA ACE modules and it hasn't come back up. The logs on the 6500 show the following..
Mar 23 08:54:25: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 08:54:28: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 08:54:28: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 08:54:27: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 08:54:28: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 08:54:43: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 08:55:18: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 08:57:30: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:07:23: %SNMP-5-MODULETRAP: Module 4 [Down] Trap
Mar 23 09:07:23: SP: The PC in slot 4 is shutting down. Please wait ...
Mar 23 09:07:56: SP: PC shutdown completed for module 4
Mar 23 09:08:06: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set off (Fabric channel errors)
Mar 23 09:15:48: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 09:15:50: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 09:15:51: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 09:15:51: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 09:15:51: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 09:16:06: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:16:41: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 09:17:45: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:28:00: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:28:46: %SNMP-5-MODULETRAP: Module 4 [Down] Trap
Mar 23 09:28:46: SP: The PC in slot 4 is shutting down. Please wait ...
Mar 23 09:29:19: SP: PC shutdown completed for module 4
Mar 23 09:29:29: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set off (Fabric channel errors)
Mar 23 09:37:11: %DIAG-SP-6-RUN_COMPLETE: Module 4: Running Complete Diagnostics...
Mar 23 09:37:13: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Mar 23 09:37:13: %SNMP-5-MODULETRAP: Module 4 [Up] Trap
Mar 23 09:37:12: %DIAG-SP-6-DIAG_OK: Module 4: Passed Online Diagnostics
Mar 23 09:37:13: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
Mar 23 09:37:28: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
Mar 23 09:38:03: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Mar 23 09:38:15: %SVCLC-5-SVCLCNTP: Could not update clock on the module 4, rc is -1
The output of the ACE console is the following....
System Bootstrap, Version 12.2[123],
Copyright (c) 1994-2009 by cisco Systems, Inc.
Slot 4 : Running DEFAULT rommon image ...
.ACE platform with 1048576 Kbytes of main memory
.Loading disk0:c6ace-t1k9-mz.A2_3_4.bin. Please wait ....
Uncompressing Linux...
Starting the kernel...
INIT: version 2.78 booting
Mounting Second Ramdisk ....
Second Ramdisk successfully mounted
Configuring network interfaces.
CF dump: Register callback functions
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
/dev/cf: 11 files, 26575/63414 clusters
FAT FS is ok
Compact Flash size 1014624(in 1k blocks) ...
Core file size 204800
Available free size in cf is 589424 (in 1k blocks) ...
set_coredump 2.11, 12 Mar 2005, FAT32, LFN
first_cluster = 0x5102 num_cluster = 0x40 (64)
inserting procfs
inserting isan_kthread
inserting wiremod
inserting klib
inserting resdrv
inserting tlv
inserting sse
inserting kpss
inserting sdwrap
creating sdwrap device
inserting klm_tl
creating tl device
inserting klm_scp
inserting klm_mts
creating mts0 device
creating mtscfg0 device
inserting utaker
creating utaker0 device
creating utaker1 device
inserting sysmgr-hb
creating sysmgr-hb device
inserting modlock
creating modlock device
inserting bufmgr
inserting pkt_fifo
inserting encdec
creating encdec device
inserting pseudo
inserting drammap mod
creating drammap device
inserting ixp_dnld
creating ixp_dnld device
inserting sysdrv
creating sysdrv device
New registry installed.
INIT: Entering runlevel: 3
inserting i2c module
inserting ssa driver
inserting cde driver
inserting bf_dnld driver
inserting pfm_drv driver
inserting regaccess driver
inserting bf_nvram driver
Firmware compiled 21-Jan-11 13:14 by integ Build [25600]
ACE Daughter boards DB1 not present DB2 not present.
downloading fpga to cde 1
Read 3262454 bytes from ./cde1_core.bit
FPGA Date: 2007/12/18 Time: 14:22: 0
CDE 1 download successful
downloading fpga to cde 2
Read 2377744 bytes from ./cde2_core.bit
FPGA Date: 2007/ 8/15 Time: 20:59:47
CDE 2 download successful
FPGA Programming Done
CDE 1 revision ID 0403
CDE 2 revision ID 0402
enabling cde 0 interrupts
finished CDE setup
Configuring NP 1 Memory
Configuring NP 2 Memory
.............................
Waiting for NP 1 SRAM memory to clear...success
Downloading NP 1 Image
.............................
Waiting for NP 2 SRAM memory to clear...success
Downloading NP 2 Image
..... 0x4eef60 (5173088) bytes downloaded
..... 0x4eef60 (5173088) bytes downloaded
Loading Nitrox driver.
PCI device 177d:0002
Writing register at address 3838 with e00
size = 8108
Ctx memory range(0x0000000-0x10000000)
Cleared 262144 1024-byte blocks in 5 requests.
N2SetupMicrocode: failed; error code 3
Writing register at address 3898 with 1
N2LoadMicrocode: failed; error code 3
N2LoadMicrocode: failed; error code 3
Solved! Go to Solution.
03-23-2012 02:48 AM
Hi,
Looks like its a hardware problem. Try reseating the module and see if that helps. If it still doesn't boot have a TAC case raised for further analysis and requires a replacement.
-
Siva
03-23-2012 02:48 AM
Hi,
Looks like its a hardware problem. Try reseating the module and see if that helps. If it still doesn't boot have a TAC case raised for further analysis and requires a replacement.
-
Siva
03-23-2012 03:31 AM
Thanks Siva. Have tried the reseat but still no luck. TAC case in progress. I think I've had some back luck with these as this is the second module to have gone faulty in as many weeks.
06-13-2012 02:12 PM
Hi Mick,
Any update from TAC on this ? I have also received following logs on the switch and the active ace had recovered by itself and is now working fine.
%SVCLC-5-SVCLCNTP: Could not update clock on the module 11, rc is -1
Strange thing is the system and kernel uptimes are different ?
# show system up time
System start time: Tue Jun 12 10:41:12 2012
System uptime: 1 days, 2 hours, 24 minutes, 33 seconds
Kernel uptime: 5 days, 7 hours, 25 minutes, 33 seconds
There are no messages on the switch indicating a module reload but it has caused an outage.
Regards,
Akhtar
06-13-2012 08:03 PM
Hello Akhtar,
Can you upload the command: #show version?
Can you upload the dir core: , hopefully the ACE might have generated some core dumps which might help us to determine the failure?
Here you have a link about getting the core dumps:
Have you experienced this issue before? Did you experience this issue during a high peak of traffic?
Did you apply any change in the configuration?
#show tech-support and core dumps would help to determine if this was a hardware failure or a software defect
-----------------------------------------------------------
Jorge
12-24-2014 05:56 AM
I've the same problem. if your module has been produced between 2005 - 2010 then Cisco has field notice about that. HW on referred list has memory component problem. It should be fixed asap.
http://www.cisco.com/c/en/us/support/docs/field-notices/637/fn63782.html
01-21-2015 05:32 PM
Cisco ITD might be a good fit here:
http://blogs.cisco.com/datacenter/itd-load-balancing-traffic-steering-clustering-using-nexus-5k6k7k
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide