Issue with FPD after XR upgrade 4.3.2 to 5.3.2 on PWR-3KW-AC-V2 modules

[MATTHIAS SCHAERER]

Hi

after upgrading IOS XR with "fpd auto-upgrade" enabled I ran into issues with PWR-3KW-AC-V2. I have four of them and they seem to be in inconsistent state concerning the versions on the different FPGAs. They do not let me do manual upgrades nor do they upgrade after a reboot. They are listed as failed but still draw power, no linecards are brought down so far. The ASR still seems to work properly but the state is definitely not ok.

fpga11 on modules one to three are on V6.04, the corresponding fpga12 and fpga13 should be on 6.02. The module four is powered down currently in order to maybe have one last module at a consistent state (fpd levels 6.03/6.01).

Is there any way how to get out of this situation? Maybe a downgrade/re-upgrade procedure? Any special commands? Run commands? Or is there any known bug (I checked bug navigator but no success)?

Any hint is greatly appreciated

Mat

RP/0/RSP0/CPU0:ASRix#admin show platform 
Fri Mar 18 09:15:09.489 UTC
Node           Type                    State                Config State
-----------------------------------------------------------------------------
0/RSP0/CPU0    A9K-RSP-4G(Active)      IOS XR RUN           PWR,NSHUT,MON
0/FT0/SP       ASR-9010-FAN-V2         READY 
0/FT1/SP       ASR-9010-FAN-V2         READY 
0/0/CPU0       A9K-MOD80-TR            IOS XR RUN           PWR,NSHUT,MON
0/0/1          A9K-MPA-20X1GE          OK                   PWR,NSHUT,MON
0/1/CPU0       A9K-8T-L                IOS XR RUN           PWR,NSHUT,MON
0/PM0/0/SP     PWR-3KW-AC-V2           FAILED               PWR,NSHUT,MON
0/PM0/1/SP     PWR-3KW-AC-V2           FAILED               PWR,NSHUT,MON
0/PM1/0/SP     PWR-3KW-AC-V2           FAILED               PWR,NSHUT,MON
0/PM1/1/SP     PWR-3KW-AC-V2           FAILED               PWR,NSHUT,MON

RP/0/RSP0/CPU0:ASRix#admin show hw-module fpd location all 
Fri Mar 18 09:15:23.770 UTC

===================================== ==========================================
 Existing Field Programmable Devices
 ==========================================
                                        HW                      Current SW  Upg/
Location     Card Type                Version Type Subtype Inst   Version   Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/RSP0/CPU0  A9K-RSP-4G 1.0                   lc   fpga3   0       1.23     No 
                                              lc   fpga1   0       1.05     No 
                                              lc   fpga2   0       1.15     No 
                                              lc   cbc     0       1.03     No 
                                              lc   fpga4   0       3.08     No 
                                              lc   rommon  0       1.06     No 
--------------------------------------------------------------------------------
0/FT0/SP     ASR-9010-FAN-V2 1.0              ft   cbc     7      29.11     No 
--------------------------------------------------------------------------------
0/FT1/SP     ASR-9010-FAN-V2 1.0              ft   cbc     8      29.11     No 
--------------------------------------------------------------------------------
0/PM0/0/SP   PWR-3KW-AC-V2 1.0                pm   fpga11  13      6.04^    No 
                                              pm   fpga12  13      6.01^    Yes
                                              pm   fpga13  13      6.01^    Yes
--------------------------------------------------------------------------------
0/PM0/1/SP   PWR-3KW-AC-V2 1.0                pm   fpga11  14      6.04^    No 
                                              pm   fpga12  14      6.01^    Yes
                                              pm   fpga13  14      6.01^    Yes
--------------------------------------------------------------------------------
0/PM1/0/SP   PWR-3KW-AC-V2 1.0                pm   fpga11  17      6.04^    No 
                                              pm   fpga12  17      6.01^    Yes
                                              pm   fpga13  17      6.01^    Yes
--------------------------------------------------------------------------------
0/PM1/1/SP   PWR-3KW-AC-V2 1.0                pm   fpga11  18      6.03^    Yes
                                              pm   fpga12  18      6.01^    Yes
                                              pm   fpga13  18      6.01^    Yes
--------------------------------------------------------------------------------
0/0/CPU0     A9K-MOD80-TR 1.0                 lc   cbc      0     20.118    No 
                                              lc   fpga2    0      1.04     No 
                                              lc   fpga4    0      1.05     No 
                                              lc   rommon   0      2.13     No 
--------------------------------------------------------------------------------
0/0/1        A9K-MPA-20X1GE 1.102             spa  fpga3    1      0.08     Yes
--------------------------------------------------------------------------------
0/1/CPU0     A9K-8T-L 1.0                     lc   fpga1    0      1.03     No 
                                              lc   fpga2    0      0.11     No 
                                              lc   cbc      0      6.11     No 
                                              lc   cpld2    0      0.08     No 
                                              lc   cpld1    0      1.02     No 
                                              lc   cpld3    0      0.03     No 
                                              lc   cpld4    0      1.03     No 
                                              lc   rommon   0      1.03     No 
--------------------------------------------------------------------------------
0/1/CPU0     A9K-8T-L 1.0                     lc   fpga1    1      1.03     No 
--------------------------------------------------------------------------------
NOTES:
1. One or more FPD needs an upgrade. This can be accomplished
 using the "admin> upgrade hw-module fpd <fpd> location <loc>" CLI.
2. ^ One or more FPD will be intentionally skipped from upgrade using CLI with option "all" or during "Auto fpd".
 It can be upgraded only using the "admin> upgrade hw-module fpd <fpd> location <loc>" CLI with exact location.
RP/0/RSP0/CPU0:ASRix#

[xthuijs]

hi tj,

if it is plain IP, then the mpls/label situation wouldnt apply on the 4 lane thing.

hard to say what the stipulation is that it is gone after a reload, gives us an iffy feel right... next time you see it, would recommend to open a tac case, together we can find out what precisely is going on and define a long(er) term solution or minimally understand where it comes from.

as for the hashing, the optic (type) has no influence on that, it is merely the LC type with it's mac and npu. between mac and npu there are "sgmii" lanes for 1G for instance or similar kinds for 10G/100G (eg SERDES). the mac *may* need to loadbalance on those links towards the NPU. The NPU can derive a hash based on the "documented" algorithms based on label/service etc and define the upstremam link/path etc.

So the MAC will do some basic hash in case it has multiple lanes towards the npu,

the npu will do a hash based on ip/label info (depending) and uses that to determine the egress bundle member or ecmp path.

(check CL id 2904 sandiego 2015 and sanfran 2014 for more detail on that piece)

cheers!

xander

[T J]

Thanks Xander, I have already opened a TAC case, let's see how far we can troubleshoot to find the issue. We had this issue before also and we replaced the 100g board suggested by TAC then, even after that we are having the same issue. 

Thanks

[xthuijs]

yeah this doesnt sound like a hardware issue Tj, so if the recommendation is to replace please have this escalated by the tac engineer for dev to look into.

fpd revision levels and code + smu currency is one key thing.

I sounds like this is a 2x100 typhoon card considering you mention the 25G thing,

the LB situation could be related to the fpga6 or 7 (from show hw-module fpd) of that card so focus on that rev level and use 533 for this with all the smu's (or SP).

cheers

xander