2960S post failures if break into rom?

kyle woodhouse · ‎05-09-2011

Hi there,

I just wanted to see if anyone else has experienced this issue or can replicate it for me on their 2960S switch. I have over 20 units doing the same thing and it seems to be a software isssue but i dont see any bugs or posts on it. This is only on 2960S switches and not 2960 or 2960G units.

If i use the password reset feature to break the units into rom and then type "boot" instead of power cycling the unit, they will fail MBIST post tests. If the unit is power cycled or left to boot normally on its own, there is no issues and all post tests pass. I know MBIST is Memory Built In Self Test and was thinking maybe breaking the unit into rom disrupts those memory tests for some reason. I tried the following software and got the same results with all of the images:

122-55.SE2

122-55.SE

122-53.SE2

122-53.SE1

Logs attached are from the same switch, one with password reset procedure used and while left to boot on its own. Let me know if anyone finds more details on the issue or if you can replicate the issue on your end.

Thanks,

kyle woodhouse · ‎05-09-2011

This is what the post failure looks like on bootup:

...done Initializing flashfs.

POST: MA BIST : BeginYeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G FC 1 MBIST Unload Activity

***ERR: S88G FC 1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Sg1 MBIST Unload Activity

***ERR: S88G DP Sg1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Xg1 MBIST Unload Activity

***ERR: S88G DP Xg1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G NI 1 MBIST Unload Activity

***ERR: S88G NI 1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G FC 0 MBIST Unload Activity

***ERR: S88G FC 0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Sg0 MBIST Unload Activity

***ERR: S88G DP Sg0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Xg0 MBIST Unload Activity

***ERR: S88G DP Xg0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G NI 0 MBIST Unload Activity

***ERR: S88G NI 0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G UPB MBIST Unload Activity

***ERR: S88G UPB MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

front_end/ (directory)
extracting front_end/fe_type_4 (78476 bytes)
extracting front_end/front_end_ucode_info (43 bytes)
extracting ucode_info (77 bytes)
Waiting for Stack Master Election...
POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

Election Complete
Switch 1 booting as Master
Waiting for Port download...Complete

And once fully booted, the show post output:

Switch#sh post
Stored system POST messages:

Switch 1
---------

POST: MA BIST : Begin
***ERR: S88G FC 1 MBIST Unload Activity

***ERR: S88G FC 1 MBIST failed

***ERR: S88G DP Sg1 MBIST Unload Activity

***ERR: S88G DP Sg1 MBIST failed

***ERR: S88G DP Xg1 MBIST Unload Activity

***ERR: S88G DP Xg1 MBIST failed

***ERR: S88G NI 1 MBIST Unload Activity

***ERR: S88G NI 1 MBIST failed

***ERR: S88G FC 0 MBIST Unload Activity
***ERR: S88G FC 0 MBIST failed

***ERR: S88G DP Sg0 MBIST Unload Activity

***ERR: S88G DP Sg0 MBIST failed

***ERR: S88G DP Xg0 MBIST Unload Activity

***ERR: S88G DP Xg0 MBIST failed

***ERR: S88G NI 0 MBIST Unload Activity

***ERR: S88G NI 0 MBIST failed

***ERR: S88G UPB MBIST Unload Activity

***ERR: S88G UPB MBIST failed

POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed
POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

Switch#

Leo Laohoo · ‎05-09-2011

We've deployed almost 200+ 2960S and I've seen something similar (not exactly the same MBIST failure syntax you're seeing) just only yesterday. Once I rebooted the switch, it was fine.

Odd though. First time I've seen something like this.

Julio E. Moisa · ‎05-10-2011

have you tried to delete the flash and restore the IOS using Xmodem?

>> Marcar como útil o contestado, si la respuesta resolvió la duda, esto ayuda a futuras consultas de otros miembros de la comunidad. <<

kyle woodhouse · ‎05-10-2011

I have not tried using Xmodem to install the software because the units do boot perfectly fine when left alone. Its only when you use the mode button to get into rommon and then type boot that the issue occurs.

I did format the flash and install several images while the unit was booted though. Same problem across all 4 images i tried. When ever you use the mode button to break them into rommon and do not power cycle afterwards they will fail post. I really think its a bug but was just looking for someone else to confirm or try it on their units.

Thanks for the feedback so far.

Leo Laohoo · ‎05-10-2011

It's fine Kyle.

Wait for another few weeks because Cisco will be release the rebuild of the 58SE, 12.2(58)SE1, if you want to upgrade.

Leo Laohoo · ‎05-11-2011

Right. 12.2(58)SE1 has been released, as scheduled.

Release Notes for the Catalyst 3750, 3560, 2960-S, and 2960 Switches, Cisco IOS Release 12.2(58)SE1

Cisco IOS Release 12.2(58)SE1 and later does NOT support all the Catalyst 3750 and 3560 switches. The models listed below are NOT supported in this release. For ongoing maintenance rebuilds for these switches, use Cisco IOS Release 12.2(55)SE and later (SE1, SE2, and so on).

• WS-C3560-24TS

•WS-C3560-24PS

•WS-C3560-48PS

•WS-C3560-48TS

•WS-C3750-24PS

•WS-C3750-24TS

•WS-C3750-48PS

•WS-C3750-48TS

•WS-3750G-24T

•WS-C3750G-12

•WS-C3750G-24TS

•WS-C3750G-16TD

npjames0001 · ‎05-15-2013

I've had the same issue, a WS-C2960S-48FPS-L that had previously run 12.2(55)SE5, 12.2(53)SE2 and upon upgrading to 12.2(58)SE1 found the first boot from Rommon went through fine, unfortunately the problem came back again the next two times around.

I've attempted an IOS upgrade (first attempt at rommon boot worked, second didn't), an xmodem upgrade from within IOS, and am now attempting an xmodem from rommon. Will report back on the results.

Leo Laohoo · ‎05-15-2013

I've attempted an IOS upgrade (first attempt at rommon boot worked, second didn't), an xmodem upgrade from within IOS, and am now attempting an xmodem from rommon. Will report back on the results.

Xmodem? On a 2960S? Dude, that's very, very OLD SCHOOL. I have been using a different process for years and it cuts the time from, say, 45 minutes down to 6 minutes.

This process is called booting the IOS from USB flash (which the 2960S, 3560E/X, 3750E/X, some ISR G1 and ISR G2 support).

npjames0001 · ‎05-15-2013

Excellent, that's about two percent of the hardware I deal with, I've used the USB upload on a couple of devices, but when i'm troubleshooting an issue, i'm willing to try all methods of solving it.

Anyhoo, finally found the bug report;

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/15.0_2_se/release/notes/OL25301.html

Caveats Resolved in Cisco IOS Release 15.0(2)SE

•CSCtk12589 (Catalyst 2960-S switch)

When a Catalyst 2960S series switch is booted up, a large number of Yeti2S88gMdioWr: Unknown status for write operation messages may be written to the console. Although the error messages may substantially increase bootup time, the switch is fully functional after Cisco IOS boots.

The workaround is to power cycle the switch for a quick recovery. This is a software issue and the switch does not require replacement.

And important to note also I think, is with the below output I got the end result is status passed despite the other messages.

POST: MA BIST : BeginYeti2S88gMdioWr: Unknown status for write operation

***ERR: S88G FC 1 MBIST Unload Activity

***ERR: S88G FC 1 MBIST failed