05-09-2011 09:14 AM - edited 03-04-2019 12:19 PM
Hi there,
I just wanted to see if anyone else has experienced this issue or can replicate it for me on their 2960S switch. I have over 20 units doing the same thing and it seems to be a software isssue but i dont see any bugs or posts on it. This is only on 2960S switches and not 2960 or 2960G units.
If i use the password reset feature to break the units into rom and then type "boot" instead of power cycling the unit, they will fail MBIST post tests. If the unit is power cycled or left to boot normally on its own, there is no issues and all post tests pass. I know MBIST is Memory Built In Self Test and was thinking maybe breaking the unit into rom disrupts those memory tests for some reason. I tried the following software and got the same results with all of the images:
122-55.SE2
122-55.SE
122-53.SE2
122-53.SE1
Logs attached are from the same switch, one with password reset procedure used and while left to boot on its own. Let me know if anyone finds more details on the issue or if you can replicate the issue on your end.
Thanks,
05-09-2011 09:18 AM
This is what the post failure looks like on bootup:
...done Initializing flashfs.
POST: MA BIST : BeginYeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G FC 1 MBIST Unload Activity
***ERR: S88G FC 1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Sg1 MBIST Unload Activity
***ERR: S88G DP Sg1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Xg1 MBIST Unload Activity
***ERR: S88G DP Xg1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G NI 1 MBIST Unload Activity
***ERR: S88G NI 1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G FC 0 MBIST Unload Activity
***ERR: S88G FC 0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Sg0 MBIST Unload Activity
***ERR: S88G DP Sg0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Xg0 MBIST Unload Activity
***ERR: S88G DP Xg0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G NI 0 MBIST Unload Activity
***ERR: S88G NI 0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G UPB MBIST Unload Activity
***ERR: S88G UPB MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
POST: MA BIST : End, Status Passed
POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed
front_end/ (directory)
extracting front_end/fe_type_4 (78476 bytes)
extracting front_end/front_end_ucode_info (43 bytes)
extracting ucode_info (77 bytes)
Waiting for Stack Master Election...
POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed
POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed
POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed
Election Complete
Switch 1 booting as Master
Waiting for Port download...Complete
And once fully booted, the show post output:
Switch#sh post
Stored system POST messages:
Switch 1
---------
POST: MA BIST : Begin
***ERR: S88G FC 1 MBIST Unload Activity
***ERR: S88G FC 1 MBIST failed
***ERR: S88G DP Sg1 MBIST Unload Activity
***ERR: S88G DP Sg1 MBIST failed
***ERR: S88G DP Xg1 MBIST Unload Activity
***ERR: S88G DP Xg1 MBIST failed
***ERR: S88G NI 1 MBIST Unload Activity
***ERR: S88G NI 1 MBIST failed
***ERR: S88G FC 0 MBIST Unload Activity
***ERR: S88G FC 0 MBIST failed
***ERR: S88G DP Sg0 MBIST Unload Activity
***ERR: S88G DP Sg0 MBIST failed
***ERR: S88G DP Xg0 MBIST Unload Activity
***ERR: S88G DP Xg0 MBIST failed
***ERR: S88G NI 0 MBIST Unload Activity
***ERR: S88G NI 0 MBIST failed
***ERR: S88G UPB MBIST Unload Activity
***ERR: S88G UPB MBIST failed
POST: MA BIST : End, Status Passed
POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed
POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed
POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed
POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed
Switch#
05-09-2011 04:10 PM
We've deployed almost 200+ 2960S and I've seen something similar (not exactly the same MBIST failure syntax you're seeing) just only yesterday. Once I rebooted the switch, it was fine.
Odd though. First time I've seen something like this.
05-10-2011 07:24 AM
have you tried to delete the flash and restore the IOS using Xmodem?
05-10-2011 02:34 PM
I have not tried using Xmodem to install the software because the units do boot perfectly fine when left alone. Its only when you use the mode button to get into rommon and then type boot that the issue occurs.
I did format the flash and install several images while the unit was booted though. Same problem across all 4 images i tried. When ever you use the mode button to break them into rommon and do not power cycle afterwards they will fail post. I really think its a bug but was just looking for someone else to confirm or try it on their units.
Thanks for the feedback so far.
05-10-2011 04:45 PM
It's fine Kyle.
Wait for another few weeks because Cisco will be release the rebuild of the 58SE, 12.2(58)SE1, if you want to upgrade.
05-11-2011 02:10 AM
Right. 12.2(58)SE1 has been released, as scheduled.
Release Notes for the Catalyst 3750, 3560, 2960-S, and 2960 Switches, Cisco IOS Release 12.2(58)SE1
Cisco IOS Release 12.2(58)SE1 and later does NOT support all the Catalyst 3750 and 3560 switches. The models listed below are NOT supported in this release. For ongoing maintenance rebuilds for these switches, use Cisco IOS Release 12.2(55)SE and later (SE1, SE2, and so on).
• WS-C3560-24TS
•WS-C3560-24PS
•WS-C3560-48PS
•WS-C3560-48TS
•WS-C3750-24PS
•WS-C3750-24TS
•WS-C3750-48PS
•WS-C3750-48TS
•WS-3750G-24T
•WS-C3750G-12
•WS-C3750G-24TS
•WS-C3750G-16TD
05-15-2013 02:53 PM
I've had the same issue, a WS-C2960S-48FPS-L that had previously run 12.2(55)SE5, 12.2(53)SE2 and upon upgrading to 12.2(58)SE1 found the first boot from Rommon went through fine, unfortunately the problem came back again the next two times around.
I've attempted an IOS upgrade (first attempt at rommon boot worked, second didn't), an xmodem upgrade from within IOS, and am now attempting an xmodem from rommon. Will report back on the results.
05-15-2013 03:33 PM
I've attempted an IOS upgrade (first attempt at rommon boot worked, second didn't), an xmodem upgrade from within IOS, and am now attempting an xmodem from rommon. Will report back on the results.
Xmodem? On a 2960S? Dude, that's very, very OLD SCHOOL. I have been using a different process for years and it cuts the time from, say, 45 minutes down to 6 minutes.
This process is called booting the IOS from USB flash (which the 2960S, 3560E/X, 3750E/X, some ISR G1 and ISR G2 support).
05-15-2013 03:45 PM
Excellent, that's about two percent of the hardware I deal with, I've used the USB upload on a couple of devices, but when i'm troubleshooting an issue, i'm willing to try all methods of solving it.
Anyhoo, finally found the bug report;
Caveats Resolved in Cisco IOS Release 15.0(2)SE
•CSCtk12589 (Catalyst 2960-S switch)
When a Catalyst 2960S series switch is booted up, a large number of Yeti2S88gMdioWr: Unknown status for write operation messages may be written to the console. Although the error messages may substantially increase bootup time, the switch is fully functional after Cisco IOS boots.
The workaround is to power cycle the switch for a quick recovery. This is a software issue and the switch does not require replacement.
And important to note also I think, is with the below output I got the end result is status passed despite the other messages.
POST: MA BIST : BeginYeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G FC 1 MBIST Unload Activity
***ERR: S88G FC 1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Sg1 MBIST Unload Activity
***ERR: S88G DP Sg1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Xg1 MBIST Unload Activity
***ERR: S88G DP Xg1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G NI 1 MBIST Unload Activity
***ERR: S88G NI 1 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G FC 0 MBIST Unload Activity
***ERR: S88G FC 0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Sg0 MBIST Unload Activity
***ERR: S88G DP Sg0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G DP Xg0 MBIST Unload Activity
***ERR: S88G DP Xg0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G NI 0 MBIST Unload Activity
***ERR: S88G NI 0 MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
***ERR: S88G UPB MBIST Unload Activity
***ERR: S88G UPB MBIST failed
Yeti2S88gMdioWr: Unknown status for write operation
POST: MA BIST : End, Status Passed
05-15-2013 04:10 PM
Ok, here's the deal ...
I've been warning alot of people against using IOS versions 12.2(58)SE, 15.0(1)SE, and 15.0(2)SE.
The most stable IOS I've ever encountered is 12.2(55)SE7. Try that.
If you really have to use 15.0(X), try 15.0(2)SE2.
If you need to boot the IOS from ROMmon, the command is "boot usbflash0:filename.bin".
05-15-2013 04:23 PM
Cheers for that, the input's much appreciated.
This is going back to a client with notes on where to go from here, updating/running the switch will be on them, I'll pass this note along.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide