Firmware upgrade fails on ASR 1004

[roysm]

Hi

We have some ASR-1004 routers. I tried to upgrade from 16.05.03 to 16.12.04. When I reboot the router with 16.12.04, the router does not load. As it does, I see the following messages:

--------------------------------------

RSA Signed RELEASE Image Signature Verification Successful.

Failed to Free memory block at address 0x0000000000000000

Failed to allocate enough space for the ISO image !

Failed to Free memory block at address 0x000000003decf000
boot: error executing "boot bootflash:asr1000rpx86-universalk9.16.12.04.SPA.bin"
autoboot: boot failed, restarting...

--------------------------------------

I saw an article related to the ASR1002 that suggested this was an issue with the ROMMON code. So, I upgraded this to v.17.3(1r) but I still could not upgrade the router firmware. What is strange, is that the ROMMON upgrade does not appear to update all adapters, so not sure if this could be a problem. 

--------------------------------------------

#sh platform
Chassis type: ASR1004

Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ASR1000-2T+20X1GE ok 4d03h
0/0 BUILT-IN-2T+20X1GE ok 4d03h
R0 ASR1000-RP2 ok, active 4d03h
F0 ASR1000-ESP40 ok, active 4d03h
P0 ASR1004-PWR-AC ok 4d03h
P1 ASR1004-PWR-AC ok 4d03h

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 14011701 16.3(2r)
R0 14111801 17.3(1r)
F0 1003190E 17.3(1r)

--------------------------------------------

Is the ROMMON version on slot 0 a problem? Has anyone else had an issue when upgrading ASR firmware?

Many thanks

Roy

[Leo Laohoo]

---

@roysm wrote:

Failed to Free memory block at address 0x0000000000000000

Failed to allocate enough space for the ISO image !

Failed to Free memory block at address 0x000000003decf000

---

Don't worry about the ROMMON -- You've upgraded them and there are some components that will not be supported with higher ROMMON version and this is fine.  This "feature" is not documented anywhere. 

I am more concerned about the above error messages. 

Please console into the router and reboot.  Post the entire boot-up process because we need to know what is going on. 

Can you also post the complete output to the following command: 

dir

NOTE:  Prepare to download and downgrade the ROMMON to 16.12.(8r).  I have a hunch (and I hope I am wrong), that this could be a bug with the newly-released 17.3(1r).

[roysm]

Leo

Thanks for the reply. Good to know about the ROMMON inconsistency.

I did attempt the firmware upgrade before I upgraded the ROMMON from 16.3(2r) but it failed and I saw the same memory errors. This is why I upgraded the ROMMON to 17.3(1r).

I will arrange to reboot the router and post the results. 

Thanks
Roy

[mgrabovs · Accepted solution]

Hello Roy,

I have been doing some investigation and it really seems that this is a behaviour documented in this bug:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvm90995

You can read in the bug page the reason this happens.

Anyways, below I have an example of the workaround for such bug, that should work. Basically, we need to upgrade the ROMMON, FPGA and CPLD before ugprading to a 16.12 IOS-XE version.

To boot with 16.12 releases ROMMON should be 16.9(5r) and FPGA version should be greater than 17071402.

1.- Upgrade ROMMON to 16.9.(5r)
https://software.cisco.com/download/home/282450665/type/282046486/release/16.9(5r)

# upgrade rom-monitor filename bootflash:asr1000-rommon.169_5r_SPA.pkg all

2.- Upgrade CPLD and FPGA (I have here the example for 16.11.1, you can use a more recent version)

https://software.cisco.com/download/home/282450665/type/283425232/release/16.11.1

# upgrade hw-programmable cpld filename bootflash:asr1000rpx86-hw-programmables.16.11.01.SPA.pkg all

# upgrade hw-programmable fpga filename bootflash:asr1000rpx86-hw-programmables.16.11.01.SPA.pkg all

A power cycle is required for the upgrades to take effect.

Additional information about hardware programables can be found in "Upgrading Field Programmable Hardware Devices for Cisco ASR 1000 Series Routers"

https://www.cisco.com/c/en/us/td/docs/routers/asr1000/cpld/hw_fp_upgrade.html

3.- Boot with 16.12.4 release

configure terminal
no boot system
boot system flash harddisk:asr1000rpx86-universalk9.16.12.04.SPA.bin
end
copy running-config startup-config
reload

Let me know if this works for you.

Thanks
Marin

[roysm]

Hi Marin

I had not come across that particular bug report. That does seem to be have been the problem. 

It looks like that has worked.

- I left the ROMMON on 17.3(1r). I

- then upgraded the HW programmable devices, with 16.11.1, although it only upgraded the route-processor, as the other devices were already on the same code.

- I then booted to the new IOS-XE firmware, 16.12.04 and the router successfully rebooted this time

I have other routers of this model to upgrade in the next few weeks, so hopefully this will work for them. 

Thanks for all the help. 

Thanks

Roy

[Michal Stanczyk]

The ASR1000 RP2 upgrade scenarios and related ROMMON/CPLD dependencies and prerequisites have been discussed in the below document published on cisco.com a few months ago:

https://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregation-services-routers/216564-asr1000-rp2-cpld-and-fpga-upgrade-common.html

HTH,

Michal