03-14-2012 08:08 PM - edited 03-07-2019 05:34 AM
Hi,
has anybody experience this, Our ME3400 crashed unexpectedly and got the below errors:
meQCY0290A#sh ver
Cisco IOS Software, ME340x Software (ME340x-METROACCESSK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Sun 08-Aug-10 00:21 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02600000
ROM: Bootstrap program is ME340x boot loader
BOOTLDR: ME340x Boot Loader (ME340x-HBOOT-M) Version 12.2(44r)EY, RELEASE SOFTWARE (fc1)
meQCY0290A uptime is 7 minutes
System returned to ROM by address error at PC 0x22797A8, address 0x0
System restarted at 05:47:52 PH Thu Mar 15 2012
System image file is "flash:me340x-metroaccessk9-mz.122-55.SE.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
cisco ME-3400E-24TS-M (PowerPC405) processor (revision A0) with 131072K bytes of memory.
Processor board ID FOC1339X122
Last reset from address error
2 Virtual Ethernet interfaces
25 FastEthernet interfaces
2 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
meQCY0290A#sh log | beg Mar 15
000037: .Mar 15 05:48:59.607 PH: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed state to up
000038: .Mar 15 05:48:59.607 PH: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up
000039: .Mar 15 05:48:59.607 PH: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to up
000040: .Mar 15 05:48:59.607 PH: %LINK-3-UPDOWN: Interface FastEthernet0/3, changed state to up
000041: .Mar 15 05:48:59.607 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
000042: Mar 15 05:49:00.614 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/2, changed state to up
000043: Mar 15 05:49:00.631 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up
000044: Mar 15 05:49:00.647 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up
000045: Mar 15 05:49:00.664 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to up
000046: Mar 15 05:49:02.157 PH: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.61.11.13 Port 514 started - reconnection
000047: Mar 15 05:49:07.165 PH: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.61.15.12 Port 514 started - reconnection
000048: Mar 15 05:49:08.407 PH: %REP-4-LINKSTATUS: GigabitEthernet0/2 (segment 5) is operational
000049: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: System previously crashed with the following message:
000050: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Cisco IOS Software, ME340x Software (ME340x-METROACCESSK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)
000051: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport
000052: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Copyright (c) 1986-2010 by Cisco Systems, Inc.
000053: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Compiled Sun 08-Aug-10 00:21 by prod_rel_team
000054: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED:
000055: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
000056: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED:
000057: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: SRR0 = 0x01516AF4 SRR1 = 0x00029230 SRR2 = 0x022797A8 SRR3 = 0x00029230
000058: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x10000000
000059: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED:
000060: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: CPU Register Context:
000061: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Vector = 0x00002000 PC = 0x022797A8 MSR = 0x00029230 CR = 0x22004042
000062: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: LR = 0x0227976C CTR = 0x01F9E470 XER = 0x4000007E
000063: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R0 = 0x0227976C R1 = 0x0442F860 R2 = 0x00000000 R3 = 0x03F7565C
000064: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x0442F848 R7 = 0x00000000
000065: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0x00000000 R10 = 0x02CE0000 R11 = 0x03F75658
000066: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R12 = 0xC9C25312 R13 = 0x00110000 R14 = 0x0241C768 R15 = 0x00000000
000067: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R16 = 0x0413D854 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
000068: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R20 = 0x00000000 R21 = 0xFFFFFFFF R22 = 0x00000000 R23 = 0x00000012
000069: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R24 = 0x000000EA R25 = 0x00000000 R26 = 0x0442F914 R27 = 0x00000000
000070: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R28 = 0x00000000 R29 = 0x031A0000 R30 = 0x00000000 R31 = 0x00000000
000071: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED:
000072: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Stack trace:
000073: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: PC = 0x022797A8, SP = 0x0442F860
000074: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Frame 00: SP = 0x0442F870 PC = 0x0227976C
000075: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Frame 01: SP = 0x0442F888 PC = 0x02275434
000076: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Frame 02: SP = 0x0442F8A8 PC = 0x02275930
000077: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 03: SP = 0x0442F8C8 PC = 0x0222862C
000078: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 04: SP = 0x0442F908 PC = 0x02416D18
000079: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 05: SP = 0x0442F940 PC = 0x02417378
000080: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 06: SP = 0x0442F950 PC = 0x02423888
000081: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 07: SP = 0x0442F968 PC = 0x02424FEC
000082: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 08: SP = 0x0442F980 PC = 0x02424524
000083: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 09: SP = 0x0442F990 PC = 0x02424CD4
000084: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 10: SP = 0x0442F9B0 PC = 0x02424D90
000085: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 11: SP = 0x0442F9E8 PC = 0x02417368
000086: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 12: SP = 0x0442F9F8 PC = 0x02423888
000087: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 13: SP = 0x0442FA10 PC = 0x02424FEC
000088: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 14: SP = 0x0442FA28 PC = 0x02424524
000089: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 15: SP = 0x0442FA38 PC = 0x02424CD4
000090: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED:
meQCY0290A#
Please also take note of the start of the Crashinfo file:
011771: Mar 15 05:46:41.544 PH: %SYS-6-STACKLOW: Stack for process SSH Process running low, 0/12000
=== Start of Crashinfo Collection (05:46:41 PH Thu Mar 15 2012) ===
For image:
Cisco IOS Software, ME340x Software (ME340x-METROACCESSK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Sun 08-Aug-10 00:21 by prod_rel_team
42D6B20 41D99F4 41D9A2C
442FA10: 442FA28 2424524 0 1 42D6B20 5D 442FA38 2424CD4
442FA30: 42D6B20 5D 442FA58 2424D90 442FA98 3F75749 B63 40F44EC
442FA50: 42D6B20 0 442FA90 2417368 6 1 0 1
442FA70: 1 1 442FB40 5E 41D9A2C 41D99F4 42D6B20 1
Thanks,
Bryan
Solved! Go to Solution.
03-15-2012 03:00 PM
Hello Bryan,
this is really not a hardware issue, it is a software issue, according to the extended crashinfo file, you are hitting bug
CSCth27817 (duplicated by CSCto60047), explained in the link below:-
CSCth27817
multiple crashes due to SSH Stack Low
Symptom:
A device running IOS may reload unexpectedly with a SYS-6-STACKLOW error message seen int eh crashinfo file.
Conditions:
There must be an active ssh session to or from the router
Workaround:
None
CSCto60047
Chunk corruption crash on trying to abort "show tech" over SSH
Symptoms:
A crash occurs either due to a chunk corruption or at ssh_send_queue_data.
Conditions:
This symptom occurs under the following conditions:
- An SSH session exists between two routers.
- The show tech command is issued and then aborted.
Workaround:
There is no workaround.
This bug is fixed in 12.2(58)SE1 and later images, I would suggest upgrading the IOS image to 12.2(58)SE2 as it is more stable.
it is not fixed in 12.2(55)SE4, but i am not sure about 12.2(55)SE5.
Have a great day,
Zaid
03-14-2012 08:29 PM
Cisco IOS Software, ME340x Software (ME340x-METROACCESSK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)
It's an IOS bug alright.
The IOS version sounds like the same type as the 3750. If this is the case, I'd recommend you upgrade to either 12.2(55)SE4 or SE5.
03-14-2012 08:58 PM
Hi Leolaohoo,
I have check the bug tool kit and also the output interpreter tool and it does not lead me to a Cisco Bug.
Thanks,
Bryan
03-15-2012 01:49 PM
I have check the bug tool kit and also the output interpreter tool and it does not lead me to a Cisco Bug.
I doubt if you'll be able to find it particularly with the hex codes in the output. You'll need TAC assistance to nail down the exact bug code.
I don't see any signs of hardware failure in your output.
Besides, look at your IOS version. It's an "SE" or as I'd like to call it "SE0". It's the first train of the code. So like any typical software, the first version will always contain bug. Cisco has made some bug fixes and released them through SE1, SE2, SE3, SE4 and now SE5, hence, I recommended you upgrade to either SE4 or SE5 as one of these two versions may have fixed your crashing issue.
03-15-2012 03:00 PM
Hello Bryan,
this is really not a hardware issue, it is a software issue, according to the extended crashinfo file, you are hitting bug
CSCth27817 (duplicated by CSCto60047), explained in the link below:-
CSCth27817
multiple crashes due to SSH Stack Low
Symptom:
A device running IOS may reload unexpectedly with a SYS-6-STACKLOW error message seen int eh crashinfo file.
Conditions:
There must be an active ssh session to or from the router
Workaround:
None
CSCto60047
Chunk corruption crash on trying to abort "show tech" over SSH
Symptoms:
A crash occurs either due to a chunk corruption or at ssh_send_queue_data.
Conditions:
This symptom occurs under the following conditions:
- An SSH session exists between two routers.
- The show tech command is issued and then aborted.
Workaround:
There is no workaround.
This bug is fixed in 12.2(58)SE1 and later images, I would suggest upgrading the IOS image to 12.2(58)SE2 as it is more stable.
it is not fixed in 12.2(55)SE4, but i am not sure about 12.2(55)SE5.
Have a great day,
Zaid
03-19-2012 10:57 PM
Hi Leolaohoo,
Yup I already opened a Cisco TAC for this, It's also good that Zaid Abu also sent the bug details above same as the findings by our Cisco TAC. We willl be trying to upgrade all the ME3400 switches, unfortunately the switches are to many to be upgraded immeadiately. I will give you feedback once all are upgraded.
Thanks for all your comments and all of it are helpful.
Regards,
Bryan
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide