cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Hardware Upgrade from RSP440 to RSP880-LT

575
Views
25
Helpful
0
Comments

Prerequisites

      

The following pre-requisites are necessary for the migration from RSP440 to RSP880-LT to be successful.

  • Make sure that you have console access to the router.
  • Verify that the system is running a minimum of code release 6.2.2 for the RSP880-LT to be supported.
  • The system must have 2x RSP440 cards installed in an Active/Standby configuration.
  • Generation 1 Trident Linecards should be removed from the router as they are incompatible with RSP880-LT.
  • Generation 3 Tomahawk Linecards should be powered down or temporarily removed from the chassis before proceeding with the hardware migration. Use the “hw-module power disable location node-id” command to power the LC down temporarily.

 

The router must not have Tomahawk Linecards actively running in the chassis while attempting the upgrade process as the Ethernet Out-of-Band Channel (EOBC) link between the new RSP880-LT and the Tomahawk card will be down. This will prevent the RSP440 from failing over to the RSP880-LT and the upgrade will fail.

 

Prior to beginning the upgrade, you need to initiate a “redundancy switchover” for the Standby RSP440 (RSP1) to become the Active RP. This is necessary because the RSP0 slot will be the first slot to be upgraded.

 

After the switchover, please verify that all the redundancy groups are in the ready state and the Standby RSP is NSR- Ready. Issue the “show redundancy” command to check that the RP’s have synchronised the group information between them. This is critical for the upgrade to be graceful.

 

Please continue with the upgrade procedure only after these pre-requisites have been confirmed!

 

Upgrade Process

 

Identify the Active and Standby RSP’s on the router and verify that the redundancy groups are all in the Ready state. Do this before removing any RSP’s from the device.

 

  1. Run the show platform command to verify the active RSP:

RP/0/RSP0/CPU0:ASR9K-G-1102#sh plat

Mon Feb  3 09:44:20.533 UTC

Node            Type                      State            Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0     A9K-RSP440-TR(Active)     IOS XR RUN       PWR,NSHUT,MON

0/RSP1/CPU0     A9K-RSP440-TR(Standby)    IOS XR RUN       PWR,NSHUT,MON

0/0/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/0/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/0/1           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/1/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/1/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/1/1           A9K-MPA-20X1GE            OK               PWR,NSHUT,MON

0/2/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/2/0           A9K-MPA-1X40GE            OK               PWR,NSHUT,MON

0/2/1           A9K-MPA-1X40GE            OK               PWR,NSHUT,MON

RP/0/RSP0/CPU0:ASR9K-G-1102#

 

  1. Verify that all redundancy groups are in the ready state and the Backup RSP is NSR-Ready: (for this test, NSR is not configured)

RP/0/RSP0/CPU0:ASR9K-G-1102#sh redundancy

Mon Feb  3 09:44:44.706 UTC

Redundancy information for node 0/RSP0/CPU0:

==========================================

Node 0/RSP0/CPU0 is in ACTIVE role

Node Redundancy Partner (0/RSP1/CPU0) is in STANDBY role

Standby node in 0/RSP1/CPU0 is ready

Standby node in 0/RSP1/CPU0 is NSR-not-configured

Node 0/RSP0/CPU0 is in process group PRIMARY role

Process Redundancy Partner (0/RSP1/CPU0) is in BACKUP role

Backup node in 0/RSP1/CPU0 is ready

Backup node in 0/RSP1/CPU0 is NSR-not-configured

 

Group            Primary         Backup          Status        

---------        ---------       ---------       ---------     

v6-routing       0/RSP0/CPU0     0/RSP1/CPU0     Ready         

mcast-routing    0/RSP0/CPU0     0/RSP1/CPU0     Ready         

netmgmt          0/RSP0/CPU0     0/RSP1/CPU0     Ready         

v4-routing       0/RSP0/CPU0     0/RSP1/CPU0     Ready         

central-services 0/RSP0/CPU0     0/RSP1/CPU0     Ready         

dsc              0/RSP0/CPU0     0/RSP1/CPU0     Ready         

dlrsc            0/RSP0/CPU0     0/RSP1/CPU0     Ready   

 

 

  1. Initiate a redundancy switchover, to make RSP1 the Active RP for the router:

RP/0/RSP0/CPU0:ASR9K-G-1102#redundancy switchover

Mon Feb  3 09:47:17.425 UTC

Proceed with switchover 0/RSP0/CPU0 -> 0/RSP1/CPU0? [confirm]

Initiating switch-over.

RP/0/RSP1/CPU0:Feb  3 09:47:18.624 UTC: rmf_svr[394]: %HA-REDCON-4-FAILOVER_REQU

Reboot on ASR9006-V2 RSP3 (0x100306) in slot 0

By rmf_svr via REBOOT_CAUSE_FAILOVER (4000004)

Current time: 2020-02-03 09:47:18.742, Up time: 11m 36s

Release mastership on RSP3

Using UDMA5

Normal reboot

 

  1. Verify that RSP1 is the active RSP:

RP/0/RSP1/CPU0:ASR9K-G-1102#sh plat

Mon Feb  3 09:50:16.896 UTC

Node            Type                      State            Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0     A9K-RSP440-TR(Standby)    IOS XR RUN       PWR,NSHUT,MON

0/RSP1/CPU0     A9K-RSP440-TR(Active)     IOS XR RUN       PWR,NSHUT,MON

0/0/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/0/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/0/1           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/1/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/1/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/1/1           A9K-MPA-20X1GE            OK               PWR,NSHUT,MON

0/2/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/2/0           A9K-MPA-1X40GE            OK               PWR,NSHUT,MON

0/2/1           A9K-MPA-1X40GE            OK               PWR,NSHUT,MON

 

  1. Verify that the redundancy groups are in the ready state and the Backup RSP is NSR-Ready:

RP/0/RSP1/CPU0:ASR9K-G-1102#sh redundancy

Mon Feb  3 09:50:27.869 UTC

Redundancy information for node 0/RSP1/CPU0:

==========================================

Node 0/RSP1/CPU0 is in ACTIVE role

Node Redundancy Partner (0/RSP0/CPU0) is in STANDBY role

Standby node in 0/RSP0/CPU0 is ready

Standby node in 0/RSP0/CPU0 is NSR-not-configured

Node 0/RSP1/CPU0 is in process group PRIMARY role

Process Redundancy Partner (0/RSP0/CPU0) is in BACKUP role

Backup node in 0/RSP0/CPU0 is ready

Backup node in 0/RSP0/CPU0 is NSR-not-configured

 

Group            Primary         Backup          Status        

---------        ---------       ---------       ---------     

v6-routing       0/RSP1/CPU0     0/RSP0/CPU0     Ready         

mcast-routing    0/RSP1/CPU0     0/RSP0/CPU0     Ready         

netmgmt          0/RSP1/CPU0     0/RSP0/CPU0     Ready         

v4-routing       0/RSP1/CPU0     0/RSP0/CPU0     Ready         

central-services 0/RSP1/CPU0     0/RSP0/CPU0     Ready         

dlrsc            0/RSP1/CPU0     0/RSP0/CPU0     Ready         

dsc              0/RSP1/CPU0     0/RSP0/CPU0     Ready   

 

  1. At this point, remove RSP0, the Standby RSP440 from Slot0.
  2. Insert the new RSP880-LT into Slot0 and connect to its console port.

The boot process of the newly inserted RSP880-LT needs to be interrupted to configure the ROMMON software. This can be achieved by using the CTRL+C break sequence. You may need to use the break sequence continuously until the RSP enters the ROMMON mode. If the window is missed, you should reload the RSP again or do an OIR to enter ROMMON mode as this is a necessary step of the upgrade process. You may be presented with a menu to choose different boot options – if so, please select "boot IOS XR in 32-bit mode".

Please select the operating system and the boot device:

     1) IOS-XR (32 bit Classic XR)

     2) IOS-XR 64 bit Boot previously installed image

     3) IOS-XR 64 bit Mgmt Network boot using DHCP server

     4) IOS-XR 64 bit Mgmt Network boot using local settings (iPXE)

     (Press 'p' for more option)

Selection [1/2/3/4]: 1

Selected IOS-XR (32 bit Classic XR), Continue ? Y/N: y

 

  1. Once the RSP880-LT is in ROMMON mode, run the following commands to initiate the sync process:

rommon 1 > RSP_LINK_1G=1

rommon 2 > ALDRIN_VERSION_DISABLE=1

rommon 3 > RSP4L_ALDRIN_1G=1

rommon 4 > sync

 

  1. Run the following command to verify that the config register is set correctly:

rommon 5 > confreg

           Configuration Summary

   (Virtual Configuration Register: 0x102)

enabled are:

console baud: 9600

boot: MBI Boot

 

do you wish to change the configuration? y/n  [n]:  n

 

  1. Next, do a hard reset of the RSP880-LT module with the following command:

rommon 7 > reset -h

 

Resetting hard .......

oting Main Processor

Transferring Console     

 

At this point, the RSP880-LT will disconnect you from the ROMMON console and start the reboot process. Give the module some time to boot up and synchronize the redundancy groups with the Active RSP.

 

  11. Verify with the show platform command that the RSP880-LT is now standby:

RP/0/RSP1/CPU0:ASR9K-G-1102#sh plat

Mon Feb  3 10:14:53.883 UTC

Node            Type                      State            Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0     A9K-RSP880-LT-TR(Standby) IOS XR RUN       PWR,NSHUT,MON

0/RSP1/CPU0     A9K-RSP440-TR(Active)     IOS XR RUN       PWR,NSHUT,MON

0/0/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/0/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/0/1           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/1/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/1/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/1/1           A9K-MPA-20X1GE            OK               PWR,NSHUT,MON

0/2/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/2/0           A9K-MPA-1X40GE            OK               PWR,NSHUT,MON

0/2/1           A9K-MPA-1X40GE            OK               PWR,NSHUT,MON

 

  12. Verify that the RSP880-LT has reached NSR-ready status and all groups are in the Ready state:

RP/0/RSP1/CPU0:ASR9K-G-1102#sh redundancy

Mon Feb  3 10:15:58.930 UTC

Redundancy information for node 0/RSP1/CPU0:

==========================================

Node 0/RSP1/CPU0 is in ACTIVE role

Node Redundancy Partner (0/RSP0/CPU0) is in STANDBY role

Standby node in 0/RSP0/CPU0 is ready

Standby node in 0/RSP0/CPU0 is NSR-not-configured

Node 0/RSP1/CPU0 is in process group PRIMARY role

Process Redundancy Partner (0/RSP0/CPU0) is in BACKUP role

Backup node in 0/RSP0/CPU0 is ready

Backup node in 0/RSP0/CPU0 is NSR-not-configured

 

Group            Primary         Backup          Status        

---------        ---------       ---------       ---------     

v6-routing       0/RSP1/CPU0     0/RSP0/CPU0     Ready         

mcast-routing    0/RSP1/CPU0     0/RSP0/CPU0     Ready         

netmgmt          0/RSP1/CPU0     0/RSP0/CPU0     Ready         

v4-routing       0/RSP1/CPU0     0/RSP0/CPU0     Ready         

central-services 0/RSP1/CPU0     0/RSP0/CPU0     Ready         

dlrsc            0/RSP1/CPU0     0/RSP0/CPU0     Ready         

dsc              0/RSP1/CPU0     0/RSP0/CPU0     Ready  

 

The RSP880-LT has now synchronised software packages and NSR state information with the Active RSP (RSP440) in Slot1. The module should be listed as Standby with an IOS XR RUN state.

 

Verify the RSPs have synchronized the SNMP engine ID and SNMP ifindex-table

#more disk0:snmp/ifindex-table loc 0/rsp0/cpu0

#more disk0:snmp/ifindex-table loc 0/rsp1/cpu0

#more disk0:snmp/snmp_persist loc 0/rsp0/cpu0

#more disk0:snmp/snmp_persist loc 0/rsp1/cpu0

 

  13. Manually copy any eem scripts from RSP440 to RSP880-LT disks if applicable

The RSP880-LT is now ready to claim mastership from the RSP440 so that the second RSP880-LT module can be installed into the router. You now need to do a “redundancy switchover” between the RSP’s. Once the RSP880-LT is in the Active state, you can remove the RSP440 from Slot1 and insert the second RSP880-LT in its place.

 

  14. Perform a redundancy switchover between the Active RSP440 and Standby RSP880-LT:

RP/0/RSP1/CPU0:ASR9K-G-1102#redundancy sw

Mon Feb  3 10:17:52.088 UTC

Proceed with switchover 0/RSP1/CPU0 -> 0/RSP0/CPU0? [confirm]

Initiating switch-over.

RP/0/RSP0/CPU0:Feb  3 10:17:52.981 UTC: rmf_svr[399]: %HA-REDCON

Reboot on ASR9006-V2 RSP3 (0x100306) in slot 1

By rmf_svr via REBOOT_CAUSE_FAILOVER (4000004)

Current time: 2020-02-03 10:17:53.077, Up time: 38m 59s

Release mastership on RSP3

Using UDMA5

Normal reboot

 

  • After the switchover, remove RSP440 from Slot 1 and insert the second RSP880-LT into Slot 1:

Once the second RSP880-LT has been installed into Slot 1, the module will boot up and start synchronising all the configuration and NSR state information from the Active RSP. After this process is complete, the second RSP880-LT will transition in the IOS XR RUN state – in the Standby mode.

 

NOTE: In case newly inserted second RSP880-LT fails to boot up, park standby RSP880-LT in Rommon. Type “set” to list Rommon variables. Check for the presence of Rommon variable “RSP_LINK_1G=1”. If set, unset RSP_LINK_1G and reset card by following below steps.

rommon 1 > set

PS1=rommon ! >

?=0

IOX_ADMIN_CONFIG_FILE=

ACTIVE_FCD=1

IPXE_PREF=0

CLUSTER_RACK_ID=0

RSP_LINK_1G=1

CLUSTER_NO_BOOT=

BOOT_DEV_SEQ_CONF=

BOOT_DEV_SEQ_OPER=

RMEMVA=

BSI=0

BOOT=disk0:asr9k-os-mbi-6.4.2/0x100305/mbiasr9k-rsp3.vm,1;

 

rommon 3 > priv

You now have access to the full set of monitor commands.

Warning: some commands will allow you to destroy your

configuration and/or system images and could render

the machine unbootable.

rommon 4 > unset RSP_LINK_1G

rommon 5 > sync

rommon 6 > sync

rommon 7 > reset -h

 

Resetting hard .......

!oting Main Processor

Transferring Console

 

RP/0/RSP0/CPU0:ASR9K-G-1102#sh plat

Mon Feb  3 11:02:58.048 UTC

Node            Type                      State            Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0     A9K-RSP880-LT-TR(Active)  IOS XR RUN       PWR,NSHUT,MON

0/RSP1/CPU0     A9K-RSP880-LT-TR(Standby) IOS XR RUN       PWR,NSHUT,MON

0/0/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/0/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/0/1           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/1/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/1/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/1/1           A9K-MPA-20X1GE            OK               PWR,NSHUT,MON

0/2/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/2/0           A9K-MPA-1X40GE            OK               PWR,NSHUT,MON

0/2/1           A9K-MPA-1X40GE            OK               PWR,NSHUT,MON

 

Once the RSP’s have converged and synchronized all the NSR state information between them, you need to run Steps 11, 12, and 13 again. All the redundancy groups should be in the Ready state and the Backup RSP should be NSR-ready.

 

FPD Upgrade Procedure

 

As part of this migration process it should not be necessary to upgrade any FPD (Field Programmable Devices). However, please issue the “show hw-module fpd location all” command to verify if the RSP’s need to be upgraded. If required, the RSP880-LT modules can be upgraded using the following commands:

#admin upgrade hw-module fpd all location 0/rsp0/cpu0

#admin upgrade hw-module fpd all location 0/rsp1/cpu0

 

Post-Upgrade Steps

Since the RSP880-LT modules use a 10Gig EOBC link between them, we need to remove the ROMMON variable that was previously configured. This can be done from the command line.

  • Remove the ROMMON variable for the EOBC:

RP/0/RSP0/CPU0:ASR9K-G-1102#run nvram_rommonvar RSP_LINK_1G 0

Mon Feb  3 10:48:07.488 UTC

Set RSP_LINK_1G=0

RP/0/RSP0/CPU0:ASR9K-G-1102#run nvram_rommonvar ALDRIN_VERSION_DISABLE 0

Mon Feb  3 10:48:23.040 UTC

Set ALDRIN_VERSION_DISABLE=0

RP/0/RSP0/CPU0:ASR9K-G-1102#run nvram_rommonvar RSP4L_ALDRIN_1G 0      

Mon Feb  3 10:48:36.360 UTC

Set RSP4L_ALDRIN_1G=0

RP/0/RSP0/CPU0:ASR9K-G-1102#

  • Verify that the ROMMON variable for 1Gig EOBC has been disabled:

RP/0/RSP0/CPU0:ASR9K-G-1102#more nvram:/classic-rommon-var             

Mon Feb  3 10:48:41.460 UTC

  PS1 = rommon ! > , ACTIVE_FCD = 1, IP_ADDRESS = 172.18.31.1, IP_SUBNET_MASK = 255.248.0.0, DEFAULT_GATEWAY = 172.16.255.254, ? = 0, IPXE_PREF = 0, CLUSTER_RACK_ID = 0, TURBOBOOT = , BSI = 0, BOOT = disk0:asr9k-os-mbi-6.4.2/0x100305/mbiasr9k-rsp3.vm,1;, RMEMVA = , CLUSTER_NO_BOOT = , BOOT_DEV_SEQ_CONF = , BOOT_DEV_SEQ_OPER = , RSP_LINK_1G = 0, ALDRIN_VERSION_DISABLE = 0, RSP4L_ALDRIN_1G = 0, confreg = 0x102RP/0/RSP0/CPU0:ASR9K-G-1102#

RP/0/RSP0/CPU0:ASR9K-G-1102#

  • Verify that the RSP880-LT EOBC ports are running at 10Gig with the following commands. The output should say 10G/KR/1-lane for all the commands:

show controllers epm-switch port-status 8 loc 0/rsp0/cpu0 | in Mode

show controllers epm-switch port-status 9 loc 0/rsp0/cpu0 | in Mode

show controllers epm-switch port-status 8 loc 0/rsp1/cpu0 | in Mode

show controllers epm-switch port-status 9 loc 0/rsp1/cpu0 | in Mode

Note:  If there is no output from previous command,   it means that the port mapping is not clear. In this case, follow below steps.

show controllers epm-switch  port-mapping  location 0/RSP0/CPU0      << find the  port connected to peer RP and then use command:

 

 RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch  port-mapping  locatio$

Tue Feb  4 16:32:11.884 UTC

   Port     |    Link Status  |      Vlan     | Connected to 

------------|-----------------|---------------|---------------

        4   |            Up   |  VLAN_EOBC_0  |  LC_EOBC_0_0

        5   |            Up   |  VLAN_EOBC_0  |  LC_EOBC_1_0

        6   |            Up   |  VLAN_EOBC_0  |  LC_EOBC_2_0

        7   |          Down   |  VLAN_EOBC_0  |  LC_EOBC_3_0

        8   |            Up   |  VLAN_EOBC_0  |  PEER_RP_0_0

        9   |            Up   |  VLAN_EOBC_1  |  PEER_RP_0_1

       10   |            Up   |    VLAN_PUNT  |  TORNADO_PL_0_0

       11   |            Up   |    VLAN_PUNT  |  TORNADO_PL_1_0

       12   |            Up   |    VLAN_PUNT  |  TORNADO_PL_2_0

       13   |            Up   |    VLAN_PUNT  |  TORNADO_PL_3_0

       14   |          Down   |    VLAN_PUNT  |  TORNADO_PL_4_0

       15   |          Down   |    VLAN_PUNT  |  TORNADO_PL_5_0

       16   |          Down   |    VLAN_PUNT  |  TORNADO_PL_6_0

       17   |          Down   |    VLAN_PUNT  |  TORNADO_PL_7_0

       23   |            Up   |    VLAN_PUNT  |  NIANTIC_0_0

       24   |            Up   | VLAN_10XGE_SWITCH  |  NIANTIC_0_1

       25   |            Up   | VLAN_10XGE_SWITCH  |  10XGE SWITCH_0_0

       28   |            Up   |  VLAN_EOBC_1  |  NIANTIC_0_0

       29   |            Up   |  VLAN_EOBC_0  |  NIANTIC_0_1

       30   |          Down   |  VLAN_EOBC_0  |  OFFLOAD_0_0

       31   |          Down   |  VLAN_EOBC_0  |  OFFLOAD_1_0

----------------------------------------------------------

 

 

show controllers epm-switch port-status <port> loc 0/rsp0/cpu0   <<   where <port> is from previous output related to PEER_RP)

 

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-status 8 loc 0/rs$

Tue Feb  4 16:37:09.369 UTC

    Mode         : 10G/KR/1-lane

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-status 9 loc 0/rs$

Tue Feb  4 16:37:09.775 UTC

    Mode         : 10G/KR/1-lane

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-status 8 loc 0/rs$

Tue Feb  4 16:37:09.827 UTC

    Mode         : 10G/KR/1-lane

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-status 9 loc 0/rs$

Tue Feb  4 16:37:09.881 UTC

    Mode         : 10G/KR/1-lane

 

At this point the hardware upgrade has been completed. Please do the necessary checks and verifications to confirm that the router is fully operational and traffic patterns are as expected