3750X Error: ASIC/PHY POST failed. Cannot continue

[Ayham Dardari]

Hi dears,

I was upgrading a cisco WS-C3750X-48 from version 12.2(58)SE2 to version 15.0(2)SE8 , everything went well (IOS extract , microcode upgrade ,...) , then during the POST the following error happened and the switch kept reloading :

POST: PortASIC Port Loopback Tests : Begin

POST: Failed PortPhyLoopback Packet Test asic_index 1 port_hardware_index 19

POST: Failed PortPhyLoopback Packet Test asic_index 1 port_hardware_index 18

POST: Failed PortPhyLoopback Packet Test asic_index 1 port_hardware_index 23

POST: Failed PortPhyLoopback Packet Test asic_index 1 port_hardware_index 22

POST: PortASIC Port Loopback Tests : End, Status Failed

Error: ASIC/PHY POST failed. Cannot continue.

%Software-forced reload

I have tried to boot the IOS from the recovery mode, I made flash_init the the boot commands , but it's always facing the same error in POST .

what might be the problem please ?

[Mark Malone · Accepted solution]

is the 12 image still in flash, if you boot that from rommon does it come back 15 version may have got corrupted or hit a severe bug

A software-forced crash occurs when the router detects a severe, unrecoverable error, and reloads itself so that it does not transmit corrupted data. A vast majority of software-forced crashes are caused by Cisco IOS^® software bugs, although some platforms can report a hardware problem as a software-forced crash.

[Ayham Dardari]

yes the 12 image still in flash , and the switch can still boot by the 12 image (I have rolled back the 12 image now) .

So you suspect it might be a software bug ?

[Mark Malone]

yes if its working on 12 its not hardware unless you dont have enough dram etc to run it but itooks like its software , if it was hardwrae it wouldnt boot either , i would redownload a fresh image do an md5 check on it make sure its ok , boot switch back to version 12 get it up and running , re-ftp version 15 back to flash run the command verify md5 etc make sure its good and reload it again with boot parameter set for 15 

[concordianag]

The exact same thing occurred when I was updating my 3750x from iOS  15.0(2)SE5 to 15.0(2)SE9.  I don't have the previous iOS version on it.  It keeps looping.  I tried copying an image from USB to flash but it always fails to copy displaying an 'unknown error'.  

Any recommendations?

Thanks!

[Mark Malone]

Try break into rommom and upload by xmodem

Recovering from Corrupted Software By Using the XMODEM Protocol

Switch software can be corrupted during an upgrade, by downloading the wrong file to the switch, and by deleting the image file. In all of these cases, the switch does not pass the power-on self-test (POST), and there is no connectivity.

This procedure uses the XMODEM Protocol to recover from a corrupt or wrong image file. There are many software packages that support the XMODEM Protocol, and this procedure is largely dependent on the emulation software you are using.

This recovery procedure requires that you have physical access to the switch.

---

Step 1 From your PC, download the software image tar file (image_filename.tar) from Cisco.com.

The Cisco IOS image is stored as a bin file in a directory in the tar file. For information about locating the software image files on Cisco.com, refer to the release notes.

Step 2 Extract the bin file from the tar file.

•If you are using Windows, use a zip program that is capable of reading a tar file. Use the zip program to navigate to and extract the bin file.

•If you are using UNIX, follow these steps:

1. Display the contents of the tar file by using the tar -tvf <image_filename.tar> UNIX command.

switch% tar -tvf image_filename.tar

drwxr-xr-x 9658/25       0 Apr 21 13:20 2003 c3750-i5-mz.121.11-AX/

drwxr-xr-x 9658/25       0 Apr 18 18:31 2003 c3750-i5-mz.121.11-AX/html/

-rw-r--r-- 9658/25    4005 Apr 18 15:56 2003 
c3750-i5-mz.121.11-AX/html/homepage.htm

-rw-r--r-- 9658/25    1392 Apr 18 15:56 2003 
c3750-i5-mz.121.11-AX/html/not_supported.html

-rw-r--r-- 9658/25    9448 Apr 18 15:56 2003 c3750-i5-mz.121.11-AX/html/common.js

-rw-r--r-- 9658/25   22152 Apr 18 15:56 2003 
c3750-i5-mz.121.11-AX/html/cms_splash.gif

-rw-r--r-- 9658/25    1211 Apr 18 15:56 2003 
c3750-i5-mz.121.11-AX/html/cms_13.html

-rw-r--r-- 9658/25    2823 Apr 18 15:56 2003 
c3750-i5-mz.121.11-AX/html/cluster.html

-rw-r--r-- 9658/25    4195 Apr 18 15:56 2003 
c3750-i5-mz.121.11-AX/html/Redirect.jar

-rw-r--r-- 9658/25   14984 Apr 18 15:56 2003 
c3750-i5-mz.121.11-AX/html/mono_disc.sgz

-rw-r--r-- 9658/25  1329516 Apr 18 15:56 2003 c3750-i5-mz.121.11-AX/html/CMS.sgz

-rw-r--r-- 9658/25  140105 Apr 18 15:56 2003 c3750-i5-mz.121.11-AX/html/images.sgz

-rw-r--r-- 9658/25  213848 Apr 18 15:56 2003 c3750-i5-mz.121.11-AX/html/help.sgz

-rw-r--r-- 9658/25  135599 Apr 18 15:56 2003 
c3750-i5-mz.121.11-AX/html/CiscoChartPanel.sgz

-rwxr-xr-x 9658/25   58860 Apr 18 18:31 2003 
c3750-i5-mz.121.11-AX/html/cms_boot.jar

-rw-r--r-- 9658/25  3970586 Apr 21 12:00 2003 
c3750-i5-mz.121.11-AX/c3750-i5-mz.121.11-AX.bin

-rw-r--r-- 9658/25     391 Apr 21 13:20 2003 c3750-i5-mz.121.11-AX/info

-rw-r--r-- 9658/25      98 Apr 18 16:46 2003 info

2. Locate the bin file and extract it by using the tar -xvf <image_filename.tar> <image_filename.bin> UNIX command.

switch% tar -xvf image_filename.tar image_filename.bin

x c3750-i5-mz.121.11-AX/c3750-i5-mz.121.11-AX.bin, 3970586 bytes, 7756 tape blocks

3. Verify that the bin file was extracted by using the ls -l <image_filename.bin> UNIX command.

switch% ls -l image_filename.bin

-rw-r--r--   1 boba      3970586 Apr 21 12:00 
c3750-i5-mz.121.11-AX/c3750-i5-mz.121.11-AX.bin

Step 3 Connect your PC with terminal-emulation software supporting the XMODEM Protocol to the switch console port.

Step 4 Set the line speed on the emulation software to 9600 baud.

Step 5 Unplug the switch power cord.

Step 6 Press the Mode button, and at the same time, reconnect the power cord to the switch.

You can release the Mode button a second or two after the LED above port 1 goes off. Several lines of information about the software appear along with instructions:

The system has been interrupted prior to initializing the flash file system. The following 
commands will initialize the flash file system, and finish loading the operating system 
software#

flash_init

load_helper

boot

Step 7 Initialize the Flash file system:

switch: flash_init

Step 8 If you had set the console port speed to anything other than 9600, it has been reset to that particular speed. Change the emulation software line speed to match that of the switch console port.

Step 9 Load any helper files:

switch: load_helper

Step 10 Start the file transfer by using the XMODEM protocol.

switch: copy xmodem: flash:image_filename.bin

Step 11 After the XMODEM request appears, use the appropriate command on the terminal-emulation software to start the transfer and to copy the software image into Flash memory.

Step 12 Boot the newly-downloaded Cisco IOS image.

switch:boot flash:image_filename.bin

Step 13 Use the archive download-sw privileged EXEC command to download the software image to the switch or to the switch stack.

Step 14 Use the reload privileged EXEC command to restart the switch and to verify that the new software image is operating properly.

Step 15 Delete the flash:image_filename.bin file from the switch.

[concordianag]

Thanks Mark!

One issue I am now having is I am running out of space on switch: 

## Send (x) c3750e-ipbasek9-mz.#11936D0.bin: Cancelled ...........................................................flash:c3750e-ipbasek9-mz.150-2.SE9.bin: no space left on device

When I try to delete the folder it gives me the error: 'no such device'

switch: dir flash: Directory of flash:/ 2 -rwx 3096 <date> multiple-fs 3 -rwx 1915 <date> private-config.text 4 -rwx 14056 <date> vlan.dat 5 drwx 3072 <date> crashinfo_ext 44 -rwx 46075 <date> config.text 45 drwx 512 <date> c3750e-ipbasek9-mz.150-2.SE9 48 drwx 512 <date> c3750e-ipbasek9-mz.150-2.SE2 571 drwx 4608 <date> crashinfo 633 drwx 512 <date> info_file_dir 244224 bytes available (57427456 bytes used) switch: delete c3750e-ipbasek9-mz.150-2.SE9 Are you sure you want to delete "c3750e-ipbasek9-mz.150-2.SE9" (y/n)?y File "c3750e-ipbasek9-mz.150-2.SE9" not deleted -- no such device

[Mark Malone]

Did you try delete flash:c3750...... ? 

[concordianag]

Yes and I get an error, the error is: "

switch: delete c3750e-ipbasek9-mz.150-2.SE9 Are you sure you want to delete "c3750e-ipbasek9-mz.150-2.SE9" (y/n)?y

File "c3750e-ipbasek9-mz.150-2.SE9" not deleted -- no such device

*EDIT* ok now it says after using /force /recursive that is cannot delete because it's a directory ..

Any idea why?

[concordianag]

Ok I think I got it.  No matter what I did I could not remove the folders in flash with delete/recursive or rmdir so I 'formatted' flash and it wiped clean, now there is enough room :)

*EDIT* I cancelled this, way too slow and used my USB key instead, was real fast and it is now booting.  Thanks for all the help!  

I think formatting the flash solved all of the weird issues.

Thanks!

[Leo Laohoo]

Nope.  The appliance is dead.  The only way is to RMA.