2960S stacking error

Kynnzak · ‎08-06-2013

OK, I need some help understanding why I'm getting the following error scrolling continuously on two stacked 2960S 24 port PoE switches.

MAIN_LODGE_MDF_SWITCH#17:48:29.541 MST: platform assert failure: 0: ../src-hulc/src-common/hulc_mad_sd_mgr.c: 658: hmsm_l3_get_locked_mad (MAIN_LODGE_MDF_SWITCH-2)

026618: *Feb 28 17:48:29.541 MST: -Traceback= 11C0D78 11B1E4C 11B3328 10CF6E8 10D0824 10D282C 10D4450 10D5830 10D7510 116467C 1173F38 1168694 1164B78 115A724 115B014 18769A8 (MAIN_LODGE_MDF_SWITCH-2)

026619: *Feb 28 17:48:29.541 MST: platform assert failure: 0: ../src-hulc/src-common/hulc_mad_sd_mgr.c: 658: hmsm_l3_get_locked_mad (MAIN_LODGE_MDF_SWITCH-2)

026620:

MAIN_LODGE_MDF_SWITCH#*Feb 28 17:48:29.541 MST: -Traceback= 11C0D78 11B1E4C 11B3328 10CF6E8 10D0824 10D282C 10D4450 10D5830 10D7510 116467C 116754C 1174080 1168694 1164B78 115A724 115B014 (MAIN_LODGE_MDF_SWITCH-2)

026621: *Feb 28 17:48:29.546 MST: platform assert failure: 0: ../src-hulc/src-common/hulc_mad_sd_mgr.c: 658: hmsm_l3_get_locked_mad (MAIN_LODGE_MDF_SWITCH-2)

IOS matches on both switches:

MAIN_LODGE_MDF_SWITCH#show version

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(55)SE7, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Mon 28-Jan-13 10:28 by prod_rel_team

Image text-base: 0x00003000, data-base: 0x01B00000

ROM: Bootstrap program is Alpha board boot loader

BOOTLDR: C2960S Boot Loader (C2960S-HBOOT-M) Version 12.2(55r)SE, RELEASE SOFTWARE (fc1)

MAIN_LODGE_MDF_SWITCH uptime is 58 minutes

System returned to ROM by power-on

System image file is "flash:/c2960s-universalk9-mz.122-55.SE7/c2960s-universalk9-mz.122-55.SE7.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C2960S-24PS-L (PowerPC) processor (revision G0) with 131072K bytes of memory.

Processor board ID FOC1721X0L2

Last reset from power-on

4 Virtual Ethernet interfaces

1 FastEthernet interface

56 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 0C:68:03:35:A4:80

Motherboard assembly number : 73-11908-09

Power supply part number : 341-0393-02

Motherboard serial number : FOC17204YXW

Power supply serial number : DCA1717U3Y3

Model revision number : G0

Motherboard revision number : A0

Model number : WS-C2960S-24PS-L

Daughterboard assembly number : 73-11933-04

Daughterboard serial number : FOC17206GK5

System serial number : FOC1721X0L2

Top Assembly Part Number : 800-30945-04

Top Assembly Revision Number : A0

Version ID : V04

CLEI Code Number : COMGE00ARD

Daughterboard revision number : A0

Hardware Board Revision Number : 0x01

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 28 WS-C2960S-24PS-L 12.2(55)SE7 C2960S-UNIVERSALK9-M

2 28 WS-C2960S-24PS-L 12.2(55)SE7 C2960S-UNIVERSALK9-M

Switch 02

---------

Switch Uptime : 22 minutes

Base ethernet MAC Address : C4:0A:CB:10:E1:80

Motherboard assembly number : 73-11908-06

Power supply part number : 341-0393-02

Motherboard serial number : FOC15496YA1

Power supply serial number : LIT15410X7U

Model revision number : D0

Motherboard revision number : A0

Model number : WS-C2960S-24PS-L

Daughterboard assembly number : 73-11933-04

Daughterboard serial number : FOC15486DNP

System serial number : FOC1550X52S

Top assembly part number : 800-30945-02

Top assembly revision number : B0

Version ID : V02

CLEI Code Number : COMGE00ARB

Daughterboard revision number : A0

Configuration register is 0xF

Switches are configured with numbers and priorities appropriately:

MAIN_LODGE_MDF_SWITCH#show switch

Switch/Stack Mac Address : 0c68.0335.a480

H/W Current

Switch# Role Mac Address Priority Version State

----------------------------------------------------------

*1 Master 0c68.0335.a480 15 1 Ready

2 Member c40a.cb10.e180 14 1 Ready

And the stacking info is:

MAIN_LODGE_MDF_SWITCH#show platform stack manager all

Switch/Stack Mac Address : 0c68.0335.a480

H/W Current

Switch# Role Mac Address Priority Version State

----------------------------------------------------------

*1 Master 0c68.0335.a480 15 1 Ready

2 Member c40a.cb10.e180 14 1 Ready

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok 2 2

2 Ok Ok 1 1

Stack Discovery Protocol View

==============================================================

Switch Active Role Current Sequence Dirty

Number State Number Bit

--------------------------------------------------------------------

1 TRUE Master Ready 131 FALSE

2 TRUE Member Ready 214 FALSE

Stack State Machine View

==============================================================

Switch Master/ Mac Address Version Current

Number Member (maj.min) State

-----------------------------------------------------------

1 Master 0c68.0335.a480 1.45 Ready

2 Member c40a.cb10.e180 1.45 Ready

Last Conflict Parameters

Switch Master/ Cfgd Default Image H/W # of Mac Address

Number Member Prio Config Type Prio Members

-----------------------------------------------------------------------

1 Member 15 No 4 5 0 0c68.0335.a480

2 Member 14 No 4 5 0 c40a.cb10.e180

Stack Discovery Protocol Counters

Messages Sent Messages Recvd

UP DOWN UP DOWN

--------------------------------------------------------

*1: 0000005722 0000005726 0000005091 0000005082

2: 0000001878 0000001878 0000001926 0000001927

3: 0000000000 0000000000 0000000000 0000000000

4: 0000000000 0000000000 0000000000 0000000000

Misc Counters

Counter Up Down

---------------------------------------------------

Wrong Ver Number: Send: 0000000000 0000000000

Wrong Ver Number: Recv: 0000000000 0000000000

Missed Messages: 0000000000 0000000000

Orphaned Messages 0000000000 0000000000

Suppressed Messages 0000000000 0000000000

No Available Messages 0000000000 0000000000

Link Present 0000000000 0000000000

Link Not Present 0000000000 0000000000

Link RxReset 0000000000 0000000000

Link Sync Stuck Resets 0000000000 0000000000

Duplicates 0000000001 0000000000

RAC Not OK Resets 0000000000

Switch# of last duplicate 0000000002

Sequence Number Failures 0000000000

RAC Not OK Resets 0000000000

Sync Not OK Resets 0000000000 0000000000

Switch# of last Failure: 256 Last Difference 0

Switch Number Conflicts 0

Stack Changes 16

Int Stack Link changes 0

Int Stack Link state 0x0

Reciprocal Efficiency Changes: Upgrade 0, Downgrade 0

Resource Counters

-------------------------------------------

Chunk Alloc's 0000000016

Chunk Free's 0000000015

Enqueue Failures: 0000000000

Null Queue Failures: 0000000000

Chunk Alloc Errors: 0000000000

Stack State Machine Counters

Messages Sent Messages Recvd

-------------------------------------------------

*1: 0000000001 0000000001

2: 0000000005 0000000005

3: 0000000000 0000000000

4: 0000000000 0000000000

Any clue why this error keeps repeating itself? As far as I can tell both switches in the stack are forwarding frames appropriately.

Thanks in advance for any help,

Kevin

P.S. The only match to the error message that I could find was in cyrillic, and unfortunately I don't read Russian . . . . ;-)

Leo Laohoo · ‎08-06-2013

Firstly, it's not going to help because the switch doesn't have a valid time set. So I have no idea how far back were the "Traceback" errors generated.

Next, I've been using the same IOS you are using for months and I've never seen a Traceback like that.

I'm not saying that you are lying. I'm suggeting that it could be something wrong with the configuration of your switch or a potential harware failure.

Hmmmm ... Can you post the output to the following commands:

1. dir flash:

2. dir flash2:

dugganm · ‎07-09-2014

Hi - Did you fix this issue in the end, was it just a code upgrade, if so which level cured it - thanks?

vieuxmontreal · ‎06-05-2015

Hi - Just experienced this issue on a test stack I was preparing for production. Basically anytime I plugged a cable into any port, it started giving me many assert failures

(ex: *Mar 1 00:59:09.372: platform assert failure: 0: ../src-hulc/src-common/hulc_mad_sd_mgr.c: 658:hmsm_l3_get_locked_mad (MyRouterHostName-2) ).

As soon as I unplugged all cables, all was ok.

This was fixed by doing the following (in my case):

- I was using Static IP routing on my Master 2960S using sdm prefer lanbase-routing and ip routing. The 2nd switch in the stack wasn't in lanbase-routing mode and I believe those assertion failures came from this fact.

- I unplugged the stack cables from switch 2, logged in and set sdm prefer lanbase-routing, saved config, reloaded and then set ip routing in config mode. I saved config again. I then plugged the stack cables back in and switch automatically reloaded.

- When switch 2 came back and was added to the stack automatically, all was fine and stable. I plugged Ethernet cables in several ports without any failures and/or warning messages.

I wanted to log this on the forum because I couldn't find any solution/answer in research online.

Regards,

RvLi

Madhukrishnan Gopinathan Nair · ‎06-06-2015

This seems to be cosmetic in nature. Found one bug with matching symptoms though your IOS version does not match to it.

https://tools.cisco.com/bugsearch/bug/CSCse22370/?reffering_site=dumpcr

Cat3750 stack: hulc_mad_sd_mgr.c error messages and tracebacks
CSCse22370

Description

Symptom:

During normal operation of the switch, following error messages are generated
in the logs:

[timestamp]: platform assert failure: hms->mem_lock_count:
../src-hulc/src-common/hulc_mad_sd_mgr.c: 531: hmsm_l3_get_locked_mad
[timestamp]: -Traceback= 43AD34 42D7F4 42EEDC 34C09C 34CE58 34E8F4 3506C4
351D00 3529F8 3EA140 3ECE0C 3FC8D0 3EDF20 3EA594 3E1DE8 3E26A8

Conditions:

Normal operation of the switch.

Workaround:

None

Further Problem Description:

The traceback is cosmetic.

Hope this helps and do remember to rate post if it is found to be be useful to you.

Thanks,

Madhu