Hi,

o.panchuk · ‎01-11-2016

asr1000-rp2 after upgrade of rommon from 15.2(1r)S to 15.3(1r)S following issue persists:

Initializing Hardware ...

FPGA load not complete !!!

Calculating the ROMMON CRC ... CRC is correct

System Bootstrap, Version 15.3(1r)S, RELEASE SOFTWARE (fc1)

CPLD Version: 1 (MM/DD/YY): 09/24/13 ASR1000-RP2 Slot:0

Current image running: Boot ROM1

Last reset cause: Watchdog/ICH

Reading confreg 0x8000

Reading monitor variables from NVRAM

Enabling interrupts

Clearing CMOS RAM ...

Initializing SATA controller...done

Initializing USB host controller...done

Checking for PCIe device presence...

Did not recognize HT/FPGA. Got 0xffffffff, Expected 0x00201137

done

ASR1000-RP2 platform with 8388608 Kbytes of main memory

autoboot entry: NVRAM VALUES: bootconf: 0x0, autobootstate: 0

autobootcount: 0, autobootsptr: 0x0

Scooby PartNum read error, expected :0x7b2245, read 0x000000

Failed to read PCA8550 EEPROM DIP device

rommon 1 >

when trying to boot something:

========================

rommon 3 > boot harddisk:asr1000rp2-ipbase.03.08.01.S.153-1.S1.bin

Scooby PartNum read error, expected :0x7b2245, read 0x000000

Failed to read PCA8550 EEPROM DIP device

Warning: filesystem is not clean

File size is 0x202492f4

Located asr1000rp2-ipbase.03.08.01.S.153-1.S1.bin

Image size 539267828 inode num 12, bks cnt 131658 blk size 8*512

###################################################################################################################

Booting image with bootparam="root=/dev/ram rw console=ttyS2,9600 max_loop=64 crashkernel=128M@16M SR_BOOT=harddi"

Boot image size = 539267828 (0x202492f4) bytes

Scooby PartNum read error, expected :0x7b2245, read 0x000000

get_mac_addr: Couldn't find a valid macaddr

Image Base is: 0x000000005d1b2018

Image Size is: 0x202492f4

Package header rev 0 structure detected

Calculating SHA-1 hash...done

validate_package: SHA-1 hash:

calculated e81a464d:cd070986:06d4270b:bcf3c8d2:385e4ba7

expected e81a464d:cd070986:06d4270b:bcf3c8d2:385e4ba7

Package type:30000, flags:0x0

Parsing package TLV info:

000: 00000009:0000001d:4b45595f:544c565f - KEY_TLV_

010: 5041434b:4147455f:434f4d50:41544942 - PACKAGE_COMPATIB

020: 494c4954:59000000:00000009:0000000b - ILITY

030: 4652555f:52505f54:59504500:00000009 - FRU_RP_TYPE

040: 00000018:4b45595f:544c565f:5041434b - KEY_TLV_PACK

050: 4147455f:424f4f54:41524348:00000009 - AGE_BOOTARCH

060: 0000000e:41524348:5f693638:365f5459 - ARCH_i686_TY

070: 50450000:00000009:00000014:4b45595f - PE KEY_

080: 544c565f:424f4152:445f434f:4d504154 - TLV_BOARD_COMPAT

090: 00000009:0000000e:424f4152:445f7270 - BOARD_rp

0a0: 325f5459:50450000:00000009:00000018 - 2_TYPE

0b0: 4b45595f:544c565f:43525950:544f5f4b - KEY_TLV_CRYPTO_K

0c0: 45595354:52494e47:00000009:00000004 - EYSTRING

TLV(0): T=9, L=29, V=KEY_TLV_PACKAGE_COMPATIBILITY

FRU Key detected

TLV(1): T=9, L=11, V=FRU_RP_TYPE

found package fru type FRU_RP_TYPE

TLV(2): T=9, L=24, V=KEY_TLV_PACKAGE_BOOTARCH

ARCH Key detected

TLV(3): T=9, L=14, V=ARCH_i686_TYPE

found package arch type ARCH_i686_TYPE

TLV(4): T=9, L=20, V=KEY_TLV_BOARD_COMPAT

TLV(5): T=9, L=14, V=BOARD_rp2_TYPE

TLV(6): T=9, L=24, V=KEY_TLV_CRYPTO_KEYSTRING

TLV(7): T=9, L=4, V=none

skipping unknown TLV type 201326592

Image validated

%IOSXEBOOT-4-FILESYS_ERRORS_CORRECTED: (rp/0): bootflash 1 contained errors which were auto-corrected.

%IOSXEBOOT-4-FILESYS_ERRORS_CORRECTED: (rp/0): harddisk 1 contained errors which were auto-corrected.

%IOSXEBOOT-4-BOOT_ACTIVITY_LONG_TIME: (rp/0): prep_storage took: 154 seconds, expected max time 10 seconds

%IOSXEBOOT-4-BOOT_SRC: (rp/0): HD Boot

%IOSXEBOOT-4-BOOT_ACTIVITY_LONG_TIME: (rp/0): platform_specific_mount_media took: 155 seconds, expected max time 3s

----------------------

Starting IOS XE kernel dump from previous failure.

Expected completion time is less than 5 minutes.

System will automatically reboot within 20 minutes.

----------------------

and then it goes into boot loop.

I've never seen such an issue on ASR1000 platform(both rp1(yes I know it different) and rp2), also MD5 sums was verified for new rommon image and for IOS bin image.

Is there any thoughts why it happens? and how to fix it?

PS many thanks in advance for your time

Philip D'Ath · ‎01-11-2016

Perhaps try going to the latest in the release chain first, 15.2(4r)S, and then do the upgrade to 15.3(1r)S.

o.panchuk · ‎01-12-2016

Agree - maybe there is some upgrade procedure which could be performed properly only if upgrading in such way.

Unfortunately I do not have any chance to test it now - RP does not booting into IOS-XE - so I could not make any manipulations with rommon version. All docs describes how to upgrade/downgrade rommon from running IOS-XE. Or is there any way how to upgrade/downgrade common without IOS-XE ? or just revert to previous rommon?

Philip D'Ath · ‎01-11-2016

Have you tried removing power from the unit and then plugging it back in, rather than just a reload? it is quite a low level issue.

o.panchuk · ‎01-12-2016

Tried it today - nothing changed - issue persist.

Philip D'Ath · ‎01-12-2016

I think that is a very bad sign. I think it is time to contact TAC. I'm thinking you have an actual hardware issue.

You were really unlucky. I don't think I have ever had an issue like that on ROMMON upgrade.

o.panchuk · ‎01-12-2016

Thanks Philip,

Another one idea from local Cisco representative - failed ESP. He found that Linux kernel could not load or initialise esi_conn kernel module, then we reinstalled all modules into new chassis(to check if there is no something in between RP and ESP) - but all the same.

Now waiting for new ESP(and RP, just in case) - should come in 1.5-2 weeks and test it all again.

o.panchuk · ‎01-29-2016

Tested it all today. Issue is definitely somewhere in RP. Just left to determine if it software or hardware issue.

Vinit Jain · ‎01-29-2016

Hello,

have you tried moving back to the old IOS XE image?

Also, before you performed the upgrade, was there netflow enabled by any chance on the mgmt interface?

i see file system errors, but the linux kernel should be able to clean up most of the file system errors (for which you see the log messages).

I would try moving back to the old IOS XE image. Also, any reason why you upgrading to XE 3.8 release and not the later releases?

Regards

Vinit

Thanks
--Vinit

o.panchuk · ‎01-29-2016

Hi,

3.8 XE is an old image. When booting with new one - the issue is the same.

No, netflow was not enabled - just default config.

Thanks,

asr1000-rp2 "FPGA load not complete" and other issues after rommon upgrade