4507r, Why the supervisor engine doesn’t shift automatically to the good one

mohsin1721 · ‎11-13-2013

Hi,

We have one core switch "WS-C4507R+E" in our data centre.

It is having 2 supervisor engine "WS-X45-SUP6L-E".

Today suddenly all remote sites were not reachable.

So, i checked for routing, i checked "show ip route"

I tried every thing, after that i tried "redundancy force-switchover"

So, the active sup restarted and standby became active.

Every thing is working fine now.

So, therefore it seems there is a problem with supervisor engine.

My question is

why the supervisor engine doesn’t shift automatically to the good one?

rfalconer.sffcu · ‎11-13-2013

What does your redundancy configuration look like on the switch?

mohsin1721 · ‎11-16-2013

Dear Robert,

Redundancy config in show run is

redundancy

mode sso

!

The output of show redundancy

Switch#sh redundancy

Redundant System Information :

------------------------------

Available system uptime = 7 weeks, 2 days, 11 hours, 48 minutes

Switchovers system experienced = 1

Standby failures = 0

Last switchover reason = user forced

Hardware Mode = Duplex

Configured Redundancy Mode = Stateful Switchover

Operating Redundancy Mode = Stateful Switchover

Maintenance Mode = Disabled

Communications = Up

Current Processor Information :

-------------------------------

Active Location = slot 4

Current Software state = ACTIVE

Uptime in current state = 7 weeks, 2 days, 11 hours, 47 minutes

Image Version = Cisco IOS Software, Catalyst 4500 L3 Switch Sof

tware (cat4500e-IPBASE-M), Version 15.1(2)SG, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 05-Dec-12 05:55 by prod_rel_team

BOOT = bootflash:cat4500e-ipbase-mz.151-2.SG.bin,12;

Configuration register = 0x2102

Peer Processor Information :

----------------------------

Standby Location = slot 3

Current Software state = STANDBY HOT

Uptime in current state = 3 days, 16 minutes

Image Version = Cisco IOS Software, Catalyst 4500 L3 Switch Sof

tware (cat4500e-IPBASE-M), Version 15.1(2)SG, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 05-Dec-12 05:55 by prod_rel

BOOT = bootflash:cat4500e-ipbase-mz.151-2.SG.bin,12;

Configuration register = 0x2102

Output of show version

Switch#sh version

Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-IPBASE-M), Versio

n 15.1(2)SG, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 05-Dec-12 05:55 by prod_rel_team

ROM: 12.2(44r)SG10

Switch uptime is 7 weeks, 2 days, 12 hours, 4 minutes

Uptime for this control processor is 7 weeks, 2 days, 12 hours, 5 minutes

System returned to ROM by SSO Switchover

System image file is "bootflash:cat4500e-ipbase-mz.151-2.SG.bin"

Darkside Revision 4, Nexu Revision 14, Fortooine Revision 1.40

Last reload reason: power-on

cisco WS-C4507R+E (MPC8548) processor (revision 13) with 524288K bytes of memory

.

Processor board ID FXS1706X09M

MPC8548 CPU at 1GHz, Supervisor 6L-E

Last reset from PowerUp

19 Virtual Ethernet interfaces

80 Gigabit Ethernet interfaces

10 Ten Gigabit Ethernet interfaces

511K bytes of non-volatile configuration memory.

Configuration register is 0x2102

I am not able to get show tech-support.

It gives me details below when i type command show tech-support

Switch#sh tech-support

------------------ Mempool statistics ------------------

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 15D52B6C 439014544 234328256 204686288 198933404 195259584

-------------- Top 100 allocator pc summary -----------

Allocator PC Summary for: Processor

Displayed first 2048 Allocator PCs only

PC Total Count Name

0x122418D8 31360888 202 RF Slave Main Thread

0x1399DBAC 19025928 18 TSM Block

0x122FE824 18350444 7 Init

0x1036AFBC 17710192 15111 *Packet Header*

0x121FE328 10280156 299 Process Stack

0x1245F170 6181040 2 Init

0x1223D848 2295984 4 GaliosObflLogger_helper

0x1399FFBC 2130772 7 Init

0x12243F80 2097220 1 *Init*

0x1036B020 1849120 653 *Packet Data*

0x1250C39C 1835172 1 Init

0x1292A3F8 1734456 6 Routekernel Process

0x11F384D4 1662536 26 XOS_MEM_UTILS

0x1122D6FC 1573920 24 IPC Message Hea

0x125734A4 1359732 1 Init

0x12828888 1327156 1 Init

0x121F4DBC 1171648 16384 Init

0x137406B8 1163264 4096 Init

0x12269410 1115280 1 Init

0x137E6CA4 1048712 1 Init

0x125F3318 1048628 1 Init

0x12645570 1048628 1 Init

0x121F4D50 983288 2 Cat4k Mgmt HiPri

0x107ED8A8 903020 2 session

0x125E0FF0 786484 1 Routekernel Process

0x1122D888 774328 3410 IPC Name String

0x139C8BF4 764876 413 Init

0x10196124 720140 2 mfib_linecard

0x12598F50 704632 2 Init

0x11E408EC 665972 55 TW Buckets

0x13740678 662076 477 Init

0x10E83FEC 655836 10 AAA AttrL Sub

0x137386FC 594720 3304 Routekernel Process

0x11187058 565508 6 RFSS_server_action

0x10C8B714 560140 2 mfib_ipv6

0x113E8838 560140 2 mfib_ipv4

0x139B6A38 542764 81 Init

0x125473D8 524400 2 Init

0x125E4994 524340 1 Init

0x10EEEFF8 481808 72 Standby service handler process

0x122452A0 462236 9 Init

0x102BE24C 418412 4831 State Machine debug data

0x10377EF4 406348 113 *Hardware IDB*

0x113B15C0 400140 2 mfib_events

0x101D767C 394028 4140 IPC Session

0x1015B020 362400 2476 IDC ENTRY

0x1036CC34 361912 12 pak subblock chunk

0x102BE1E0 335004 4831 State Machine debug meta data

0x139699C4 315052 1 Init

0x1249610C 314348 1 Init

0x10ECE560 276480 3 event_trace_tbs

0x123DB288 262392 1 Init

0x10E489F8 254004 347 AAA General DB

0x124D6B80 246780 15 Init

0x124EBE18 238196 1 Init

0x11E8A824 234132 297 Process

0x139A09E4 232996 1533 RF Slave Main Thread

0x11E27620 232140 2 tunnel

0x1276CB80 229804 787 Init

0x11C98754 221372 1 Spanning Tree Opt Port Block

0x1246F600 219068 1 Init

0x121F48C4 216044 1 Init

0x10BE0800 204000 3000 *Init*

0x1015B1A4 200720 2476 IDC ENTRY DATA

0x1015B074 198648 2476 audit_sid entry

0x1096EA30 197256 6 DSensor 64

0x11CA6508 196740 3 STP Port Contro

0x125D2E34 196660 1 Init

0x111F2E68 192276 21 IPC Port

0x103FC254 187656 347 Auth Manager Context

0x124961D4 182548 1 Init

0x122A1E08 165040 1 Init

0x11F32718 161672 10 Redundancy Event Chunks

0x107ED930 158980 2 platf

0x10377EBC 157748 113 *Software IDB*

0x11683BA8 154400 8 ADJ: adjacency

0x107ED9BC 150620 2 table

0x1265780C 147548 1 Init

0x1373F42C 147028 413 Routekernel Process

0x10AAF498 140140 2 checkpoint

0x10BCBEC4 140140 2 redundancy

0x10CADF90 131500 4 AAA Huge Chunk

0x12496178 131396 1 Init

0x137EA3A0 131208 1 Init

0x12598F38 131184 2 Init

0x1037462C 131116 1 *Init*

0x10E46240 128992 348 AAA Interface Struct

0x137ED120 128136 1 Init

0x11E4BDE4 123404 591 RF Slave Main Thread

0x115E5AF4 115384 8 CEF: fib

0x10AD3D1C 109344 24 EEM ED Identity

0x10319ECC 109196 6 XDR:BufPool

0x104F2934 105400 62 IGMPSN L2MCM

0x120BAB64 102756 304 RF Slave Main Thread

0x10E60678 102488 4 AAA Acct DB chu

0x12458730 98356 1 Init

0x124587E0 98356 1 Init

0x137FA6F4 81972 1 Init

0x121F4870 80044 1 Init

0x12269434 76668 1 Init

So, I am not able to understand it is a hardware problem or problem with the IOS.

We have 1 year smartnet contract for this switch.

Talha Ansari · ‎11-16-2013

Hi,

Not sure if that was the problem was with the Supervisor Engine. You see SSO mode provides stateful redundancy for Layer 2 traffic ONLY. When a SSO failover happens the routing protocols are reset. It may be possible that reset of routing information during the force failover has caused your issue to resolve.

Hope that helps.

Regards,

Talha Ansari.

Karthick Murugan · ‎11-17-2013

Mohsin,

The switchover will happen only if they there is a physical failure/reload of the active supervisor. I believe in your case all the control-packets were dropped by the active supervisor and the supervisor failover restored the situation.

I strongly suspect that the you are hitting this bug#CSCuj73571

Hope it helps!!

Thanks

Karthick Murugan

CCIE#39285

Thanks & Regards, Karthick Murugan CCIE#39285

mohsin1721 · ‎11-17-2013

Thanks Karthick,

I will check this.

yesterday there was same problem, so i need to shift the supervisor engine again.

In 5 days this is second time.

So, there may be problem with the IOS.

I will upgrade the IOS now.

Thanks.

Karthick Murugan · ‎11-17-2013

I personally resolved many similar issues by upgrading the code. I strongly believe the issue is should be resolved after upgrade(Fixed code)

Thanks & Regards,
Karthick Murugan
CCIE#39285

Thanks & Regards, Karthick Murugan CCIE#39285

iancalderbank · ‎02-07-2014

this is an evil bug. I have a 4510/sup8E that just suddenly stopped being a layer 3 switch, tac are pointing me to this bug CSCuj73571. My box fails to arp properly on SVI interfaces - which effectively makes it useless for its main functionality! and it doesn't failover RPs to fix itself either.

whats more there is no fixed ios for sup8E! this is the flagship sup for this box, why no ios update since august 2013?

I wouldn't use a sup8E for ANYTHING until this is fixed.

Ian

Devavrat Oka · ‎02-22-2014

We are seeing the same problem on our 4510 running Sup 7-E with 03.05.00. We downgraded to 03.04.03 but still see the same issue.

The command 'show platform cpu packet drivers' shows the count stop at 2512 on queue 2. Does not increment after that.