FPD 101 for the ASR9k

Bryan Garland · ‎09-08-2016

FPD 101 for the ASR9k
Basics about FPDs
Parallel Upgrade
To reload or not to reload?
Upgrading FPDs

FPD 101 for the ASR9k

The purpose of this document is to give guidance and explain how the FPD process works for the ASR9k platform.

Basics about FPDs

Background: FPD stands for "Field Programmable Device", these are also known as "FPGAs". Sparing lots of details: FPDs are less flexible than general purpose CPUs but more flexible than completely integrated ASICs, and they can do lots of different things at a relatively high performance.

FPDs on the ASR 9000: The ASR9k utilizes a number of FPDs on both the linecards and the RSPs, and it's vitally important that the firmware versions on these FPDs are matched to the version of IOS XR that's running on the system. Running a downrev version, that is, using XR 6, but with the FPD revisions of 5 is not recommended and should be avoided. The other way around, running uprev, eg FPD of XR 6 but you are running version XR 5 is not a problem.

How do I upgrade the FPDs on my system? Easy. "upgrade hw-module fpd ..." from admin mode. Most of the time, just use the "all" commands to upgrade everything that needs it.

Why does it require a reload? Because the FPDs can only run one "image" at a time, to load new firmware they have to be completely reset. In XR (at least for the short and medium term) this means reloading the whole LC.

Why does it take so long to upgrade the FPDs themselves? It varies from chip to chip, but the data path that we use to write new firmware into the FPD itself isn't very wide/fast, so it can take a anywhere from a few seconds to a few minutes to upgrade the images.

Parallel Upgrade

Depending on release in use, the capability of a parallel upgrade for all FPD's and all linecards in one go can be used to decrease overall duration of the programming.

To reload or not to reload?

Cards will continue to run fine during the FPD update. the FPD's have a active mask and new mask, we upgrade the non active mask and use a reload to take it to the running mask.

The FPD update can be done ahead of the actual maintenance window.

In order to leverage the new FPD firmware, the card needs to be reloaded. A full chassis reload obviously will take care of that and is the easy way out.

If you update your LC FPD's, after the update completed, you can also manually kick the LC. This way if you're dual homed you're not incurring a full service outage.

However it requires some manual orchestration to reload the linecards and making sure that eventually all that have been upgraded are indeed reloaded.

For the RSP's a possibility is to do a standby RSP reload, wait for it to come back up and do a failover and repeat the standby reload (generally part of the failover/switchover). Especially when using NSR this should be seamless also.

Again this requires a bit of orchestration but if down time is of importance, this could be a solution.

Upgrading FPDs

Now when it comes to upgrading FPDs there are a couple of ways. There is a manual way to upgrade the FPDs on a per location or all locations through the CLI. There is also a configuration knob to allow the FPDs to get upgraded during a code upgrade. When upgrading FPDs, most require a reload of the linecard for the new image to take affect.

Upgrading FPDs Manually

You can see which FPDs need upgraded with the command "admin show hw-module fpd location all". Take the below output as an example. It will list all the locations and identify which ones need upgraded, if any. Now this should also be done if a new card is inserted or a card is replaced due to RMA as well. One thing to note. There is no need to downgrade FPDs. FPDs are backward compatible so there is no need to downgrade the FPDs. In fact as of 5.3.2 we no longer report "yes" in the below output if a card is higher version than the running code.

RP/0/RSP0/CPU0:ASR9010-B#admin show hw-module fpd location all
Thu Sep 8 05:19:35.074 EST

===================================== ==========================================
 Existing Field Programmable Devices
 ==========================================
                     HW                      Current SW Upg/
Location Card Type Version Type Subtype Inst Version    Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/RSP0/CPU0 A9K-RSP880-SE 1.0 lc cbc 0 34.38 No
 lc rommon 0 10.48 No
 lc fpga2 0 0.40 No
 lc fsbl 0 1.80 No
 lc lnxfw 0 1.80 No
 lc fpga3 0 0.10 No
 lc fpga4 0 0.11 No
 lc fpga5 0 0.08 No
 lc fpga6 0 0.04 No
--------------------------------------------------------------------------------
0/FT0/SP ASR-9010-FAN-V2 1.0 ft cbc 7 29.11 No
--------------------------------------------------------------------------------
0/FT1/SP ASR-9010-FAN-V2 1.0 ft cbc 8 29.11 No
--------------------------------------------------------------------------------
0/PM0/0/SP PWR-6KW-AC-V3 1.0 pm fpga11 13 4.01^ No
 pm fpga12 13 4.00^ No
 pm fpga13 13 4.01^ No
--------------------------------------------------------------------------------
0/PM0/1/SP PWR-6KW-AC-V3 1.0 pm fpga11 14 4.01^ No
 pm fpga12 14 4.00^ No
 pm fpga13 14 4.01^ No
--------------------------------------------------------------------------------
0/PM0/2/SP PWR-6KW-AC-V3 1.0 pm fpga11 15 4.01^ No
 pm fpga12 15 4.00^ No
 pm fpga13 15 4.01^ No
--------------------------------------------------------------------------------
0/PM1/0/SP PWR-6KW-AC-V3 1.0 pm fpga11 17 4.01^ No
 pm fpga12 17 4.00^ No
 pm fpga13 17 4.01^ No
--------------------------------------------------------------------------------
0/PM1/1/SP PWR-6KW-AC-V3 1.0 pm fpga11 18 4.01^ No
 pm fpga12 18 4.00^ No
 pm fpga13 18 4.01^ No
--------------------------------------------------------------------------------
0/PM1/2/SP PWR-6KW-AC-V3 1.0 pm fpga11 19 4.01^ No
 pm fpga12 19 4.00^ No
 pm fpga13 19 4.01^ No
--------------------------------------------------------------------------------
0/RSP1/CPU0 A9K-RSP880-SE 1.0 lc cbc 0 34.38 No
 lc rommon 0 10.48 No
 lc fpga2 0 0.40 No
 lc fsbl 0 1.80 No
 lc lnxfw 0 1.80 No
 lc fpga3 0 0.10 No
 lc fpga4 0 0.11 No
 lc fpga5 0 0.08 No
 lc fpga6 0 0.04 No
--------------------------------------------------------------------------------
0/0/CPU0 A9K-8X100GE-SE 1.0 lc cbc 0 38.20 No
 lc rommon 0 8.30 No
 lc fpga2 0 1.20 Yes
 lc fsbl 0 1.11 Yes
 lc lnxfw 0 1.11 Yes
 lc fpga3 0 1.03 No
 lc fpga4 0 1.04 Yes
--------------------------------------------------------------------------------
0/0/CPU0 A9K-8X100GE-SE 1.0 lc fpga3 1 1.03 No
--------------------------------------------------------------------------------
NOTES:
1. One or more FPD needs an upgrade. This can be accomplished
 using the "admin> upgrade hw-module fpd <fpd> location <loc>" CLI.
2. ^ One or more FPD will be intentionally skipped from upgrade using CLI with option "all" or during "Auto fpd".
 It can be upgraded only using the "admin> upgrade hw-module fpd <fpd> location <loc>" CLI with exact location.
RP/0/RSP0/CPU0:ASR9010-B#

So in the above example we can see there is a linecard that needs upgraded. To perform that upgrade manually, you would use the command "admin upgrade hw-module fpd all location 0/0/CPU0". This will take some time and will need to be reloaded after the upgrade is complete.

RP/0/RSP0/CPU0:ASR9010-B#admin upgrade hw-module fpd all location 0/0/CPU0
Thu Sep 8 05:29:07.378 EST

***** UPGRADE WARNING MESSAGE: *****
 * This upgrade operation has a maximum timout of 160 minutes. *
 * If you are executing the cmd for one specific location and *
 * card in that location reloads or goes down for some reason *
 * you can press CTRL-C to get back the RP's prompt. *
 * If you are executing the cmd for _all_ locations and a node *
 * reloads or is down please allow other nodes to finish the *
 * upgrade process before pressing CTRL-C. *

% RELOAD REMINDER:
 - The upgrade operation of the target module will not interrupt its normal
 operation. However, for the changes to take effect, the target module
 will need to be manually reloaded after the upgrade operation. This can
 be accomplished with the use of "hw-module <target> reload" command.
 - If automatic reload operation is desired after the upgrade, please use
 the "reload" option at the end of the upgrade command.
 - The output of "show hw-module fpd location" command will not display
 correct version information after the upgrade if the target module is
 not reloaded.
NOTE: Chassis CLI will not be accessible while upgrade is in progress.
Continue? [confirm]


FPD upgrade in progress on some hardware, reload/configuration change
on those is not recommended as it might cause HW programming failure
and result in RMA of the hardware.


Starting the upgrade/download of following FPDs:
=========== ==== ======= ======= =========== =========
 Current Upg/Dng
Location Type Subtype Upg/Dng Version Version
=========== ==== ======= ======= =========== =========
0/0/CPU0 lc fpga2 upg 1.20 1.76
 lc fsbl upg 1.11 1.78
 lc lnxfw upg 1.11 1.78
 lc fpga4 upg 1.04 1.06
------------------------------------------------------
FPD upgrade in progress. Max timeout remaining 89 min.
FPD upgrade in progress. Max timeout remaining 88 min.
FPD upgrade in progress. Max timeout remaining 87 min.
FPD upgrade in progress. Max timeout remaining 86 min.
FPD upgrade in progress. Max timeout remaining 85 min.
FPD upgrade in progress. Max timeout remaining 84 min.
FPD upgrade in progress. Max timeout remaining 83 min.
FPD upgrade in progress. Max timeout remaining 82 min.
FPD upgrade in progress. Max timeout remaining 81 min.
FPD upgrade in progress. Max timeout remaining 80 min.
FPD upgrade in progress. Max timeout remaining 79 min.
FPD upgrade in progress. Max timeout remaining 78 min.
FPD upgrade in progress. Max timeout remaining 77 min.
FPD upgrade in progress. Max timeout remaining 76 min.
FPD upgrade in progress. Max timeout remaining 75 min.
FPD upgrade in progress. Max timeout remaining 74 min.
FPD upgrade in progress. Max timeout remaining 73 min.
FPD upgrade in progress. Max timeout remaining 72 min.
FPD upgrade in progress. Max timeout remaining 71 min.
Successfully upgraded fpga2 for A9K-8X100GE-SE on location 0/0/CPU0 from 1.20 to 1.76
Successfully upgraded fsbl for A9K-8X100GE-SE on location 0/0/CPU0 from 1.11 to 1.78
Successfully upgraded lnxfw for A9K-8X100GE-SE on location 0/0/CPU0 from 1.11 to 1.78
Successfully upgraded fpga4 for A9K-8X100GE-SE on location 0/0/CPU0 from 1.04 to 1.06


FPD upgrade has ended.

Upgrading FPDs with fpd auto-upgrade

FPDs can be upgraded automatically during the activation phase of the IOS-XR code upgrade process. The configuration is an Admin level config.

RP/0/RSP0/CPU0:ASR9010-B#admin show run
Thu Sep 8 06:06:03.175 EST
Building configuration...
!! IOS XR Admin Configuration 5.3.3
username root
 group root-system
 secret 5 xxxxxxxxxxxxxxxxxx
!
fpd auto-upgrade
end

FPD auto-upgrade guidelines

It is recommended to NOT use fpd auto-upgrade unless already on 5.3.4 or higher. The reason for this is the longer upgrade times that can happen. There is a high probability that the FPD upgrades will timeout due to the 90 minute timer and there will be cards that won't get upgraded before the activation of the new code and system reload. This will potentially leave some cards upgraded and some not upgraded once the system comes back up after the upgrade. Which would still require manual upgrades. So to simplify the upgrade process and avoid the if/thens we recommend doing the FPD upgrades manually after the code upgrade. Now if you have upgraded to 5.3.4 or higher, there have also been some FPD improvements to help shorten the upgrade times significantly. So after the upgrade and doing it manually should be quicker as well even though it will require a second system reload.

Special Considerations

Special consideration for FPD upgrades if you have the following linecards:

A9K-2x100GE-SE
A9K-2x100GE-TR
A9K-1x100GE-SE
A9K-1x100GE-TR

Please see this document