Prerequisites

pallu · ‎02-06-2020

Prerequisites

The following pre-requisites are necessary for the migration from RSP440 to RSP880-LT to be successful.

Make sure that you have console access to the router.
Verify that the system is running a minimum of code release 6.2.2 for the RSP880-LT to be supported.
The system must have 2x RSP440 cards installed in an Active/Standby configuration.
Generation 1 Trident Linecards should be removed from the router as they are incompatible with RSP880-LT.
Generation 3 Tomahawk Linecards should be powered down or temporarily removed from the chassis before proceeding with the hardware migration. Use the “hw-module power disable location node-id” command to power the LC down temporarily.

The router must not have Tomahawk Linecards actively running in the chassis while attempting the upgrade process as the Ethernet Out-of-Band Channel (EOBC) link between the new RSP880-LT and the Tomahawk card will be down. This will prevent the RSP440 from failing over to the RSP880-LT and the upgrade will fail.

Prior to beginning the upgrade, you need to initiate a “redundancy switchover” for the Standby RSP440 (RSP1) to become the Active RP. This is necessary because the RSP0 slot will be the first slot to be upgraded.

After the switchover, please verify that all the redundancy groups are in the ready state and the Standby RSP is NSR- Ready. Issue the “show redundancy” command to check that the RP’s have synchronised the group information between them. This is critical for the upgrade to be graceful.

Please continue with the upgrade procedure only after these pre-requisites have been confirmed!

Upgrade Process

Identify the Active and Standby RSP’s on the router and verify that the redundancy groups are all in the Ready state. Do this before removing any RSP’s from the device.

Run the show platform command to verify the active RSP:

RP/0/RSP0/CPU0:ASR9K-G-1102#sh plat

Mon Feb 3 09:44:20.533 UTC

Node Type State Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0 A9K-RSP440-TR(Active) IOS XR RUN PWR,NSHUT,MON

0/RSP1/CPU0 A9K-RSP440-TR(Standby) IOS XR RUN PWR,NSHUT,MON

0/0/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/0/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/0/1 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/1/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/1/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/1/1 A9K-MPA-20X1GE OK PWR,NSHUT,MON

0/2/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/2/0 A9K-MPA-1X40GE OK PWR,NSHUT,MON

0/2/1 A9K-MPA-1X40GE OK PWR,NSHUT,MON

RP/0/RSP0/CPU0:ASR9K-G-1102#

Verify that all redundancy groups are in the ready state and the Backup RSP is NSR-Ready: (for this test, NSR is not configured)

RP/0/RSP0/CPU0:ASR9K-G-1102#sh redundancy

Mon Feb 3 09:44:44.706 UTC

Redundancy information for node 0/RSP0/CPU0:

==========================================

Node 0/RSP0/CPU0 is in ACTIVE role

Node Redundancy Partner (0/RSP1/CPU0) is in STANDBY role

Standby node in 0/RSP1/CPU0 is ready

Standby node in 0/RSP1/CPU0 is NSR-not-configured

Node 0/RSP0/CPU0 is in process group PRIMARY role

Process Redundancy Partner (0/RSP1/CPU0) is in BACKUP role

Backup node in 0/RSP1/CPU0 is ready

Backup node in 0/RSP1/CPU0 is NSR-not-configured

Group Primary Backup Status

--------- --------- --------- ---------

v6-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

mcast-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

netmgmt 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v4-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

central-services 0/RSP0/CPU0 0/RSP1/CPU0 Ready

dsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

dlrsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

Initiate a redundancy switchover, to make RSP1 the Active RP for the router:

RP/0/RSP0/CPU0:ASR9K-G-1102#redundancy switchover

Mon Feb 3 09:47:17.425 UTC

Proceed with switchover 0/RSP0/CPU0 -> 0/RSP1/CPU0? [confirm]

Initiating switch-over.

RP/0/RSP1/CPU0:Feb 3 09:47:18.624 UTC: rmf_svr[394]: %HA-REDCON-4-FAILOVER_REQU

Reboot on ASR9006-V2 RSP3 (0x100306) in slot 0

By rmf_svr via REBOOT_CAUSE_FAILOVER (4000004)

Current time: 2020-02-03 09:47:18.742, Up time: 11m 36s

Release mastership on RSP3

Using UDMA5

Normal reboot

Verify that RSP1 is the active RSP:

RP/0/RSP1/CPU0:ASR9K-G-1102#sh plat

Mon Feb 3 09:50:16.896 UTC

Node Type State Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0 A9K-RSP440-TR(Standby) IOS XR RUN PWR,NSHUT,MON

0/RSP1/CPU0 A9K-RSP440-TR(Active) IOS XR RUN PWR,NSHUT,MON

0/0/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/0/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/0/1 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/1/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/1/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/1/1 A9K-MPA-20X1GE OK PWR,NSHUT,MON

0/2/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/2/0 A9K-MPA-1X40GE OK PWR,NSHUT,MON

0/2/1 A9K-MPA-1X40GE OK PWR,NSHUT,MON

Verify that the redundancy groups are in the ready state and the Backup RSP is NSR-Ready:

RP/0/RSP1/CPU0:ASR9K-G-1102#sh redundancy

Mon Feb 3 09:50:27.869 UTC

Redundancy information for node 0/RSP1/CPU0:

==========================================

Node 0/RSP1/CPU0 is in ACTIVE role

Node Redundancy Partner (0/RSP0/CPU0) is in STANDBY role

Standby node in 0/RSP0/CPU0 is ready

Standby node in 0/RSP0/CPU0 is NSR-not-configured

Node 0/RSP1/CPU0 is in process group PRIMARY role

Process Redundancy Partner (0/RSP0/CPU0) is in BACKUP role

Backup node in 0/RSP0/CPU0 is ready

Backup node in 0/RSP0/CPU0 is NSR-not-configured

Group Primary Backup Status

--------- --------- --------- ---------

v6-routing 0/RSP1/CPU0 0/RSP0/CPU0 Ready

mcast-routing 0/RSP1/CPU0 0/RSP0/CPU0 Ready

netmgmt 0/RSP1/CPU0 0/RSP0/CPU0 Ready

v4-routing 0/RSP1/CPU0 0/RSP0/CPU0 Ready

central-services 0/RSP1/CPU0 0/RSP0/CPU0 Ready

dlrsc 0/RSP1/CPU0 0/RSP0/CPU0 Ready

dsc 0/RSP1/CPU0 0/RSP0/CPU0 Ready

At this point, remove RSP0, the Standby RSP440 from Slot0.
Insert the new RSP880-LT into Slot0 and connect to its console port.

The boot process of the newly inserted RSP880-LT needs to be interrupted to configure the ROMMON software. This can be achieved by using the CTRL+C break sequence. You may need to use the break sequence continuously until the RSP enters the ROMMON mode. If the window is missed, you should reload the RSP again or do an OIR to enter ROMMON mode as this is a necessary step of the upgrade process. You may be presented with a menu to choose different boot options – if so, please select "boot IOS XR in 32-bit mode".

Please select the operating system and the boot device:

1) IOS-XR (32 bit Classic XR)

2) IOS-XR 64 bit Boot previously installed image

3) IOS-XR 64 bit Mgmt Network boot using DHCP server

4) IOS-XR 64 bit Mgmt Network boot using local settings (iPXE)

(Press 'p' for more option)

Selection [1/2/3/4]: 1

Selected IOS-XR (32 bit Classic XR), Continue ? Y/N: y

Once the RSP880-LT is in ROMMON mode, run the following commands to initiate the sync process:

rommon 1 > RSP_LINK_1G=1

rommon 2 > ALDRIN_VERSION_DISABLE=1

rommon 3 > RSP4L_ALDRIN_1G=1

rommon 4 > sync

Run the following command to verify that the config register is set correctly:

rommon 5 > confreg

Configuration Summary

(Virtual Configuration Register: 0x102)

enabled are:

console baud: 9600

boot: MBI Boot

do you wish to change the configuration? y/n [n]: n

Next, do a hard reset of the RSP880-LT module with the following command:

rommon 7 > reset -h

Resetting hard .......

oting Main Processor

Transferring Console

At this point, the RSP880-LT will disconnect you from the ROMMON console and start the reboot process. Give the module some time to boot up and synchronize the redundancy groups with the Active RSP.

11. Verify with the show platform command that the RSP880-LT is now standby:

RP/0/RSP1/CPU0:ASR9K-G-1102#sh plat

Mon Feb 3 10:14:53.883 UTC

Node Type State Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0 A9K-RSP880-LT-TR(Standby) IOS XR RUN PWR,NSHUT,MON

0/RSP1/CPU0 A9K-RSP440-TR(Active) IOS XR RUN PWR,NSHUT,MON

0/0/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/0/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/0/1 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/1/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/1/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/1/1 A9K-MPA-20X1GE OK PWR,NSHUT,MON

0/2/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/2/0 A9K-MPA-1X40GE OK PWR,NSHUT,MON

0/2/1 A9K-MPA-1X40GE OK PWR,NSHUT,MON

12. Verify that the RSP880-LT has reached NSR-ready status and all groups are in the Ready state:

RP/0/RSP1/CPU0:ASR9K-G-1102#sh redundancy

Mon Feb 3 10:15:58.930 UTC

Redundancy information for node 0/RSP1/CPU0:

==========================================

Node 0/RSP1/CPU0 is in ACTIVE role

Node Redundancy Partner (0/RSP0/CPU0) is in STANDBY role

Standby node in 0/RSP0/CPU0 is ready

Standby node in 0/RSP0/CPU0 is NSR-not-configured

Node 0/RSP1/CPU0 is in process group PRIMARY role

Process Redundancy Partner (0/RSP0/CPU0) is in BACKUP role

Backup node in 0/RSP0/CPU0 is ready

Backup node in 0/RSP0/CPU0 is NSR-not-configured

Group Primary Backup Status

--------- --------- --------- ---------

v6-routing 0/RSP1/CPU0 0/RSP0/CPU0 Ready

mcast-routing 0/RSP1/CPU0 0/RSP0/CPU0 Ready

netmgmt 0/RSP1/CPU0 0/RSP0/CPU0 Ready

v4-routing 0/RSP1/CPU0 0/RSP0/CPU0 Ready

central-services 0/RSP1/CPU0 0/RSP0/CPU0 Ready

dlrsc 0/RSP1/CPU0 0/RSP0/CPU0 Ready

dsc 0/RSP1/CPU0 0/RSP0/CPU0 Ready

The RSP880-LT has now synchronised software packages and NSR state information with the Active RSP (RSP440) in Slot1. The module should be listed as Standby with an IOS XR RUN state.

Verify the RSPs have synchronized the SNMP engine ID and SNMP ifindex-table

#more disk0:snmp/ifindex-table loc 0/rsp0/cpu0

#more disk0:snmp/ifindex-table loc 0/rsp1/cpu0

#more disk0:snmp/snmp_persist loc 0/rsp0/cpu0

#more disk0:snmp/snmp_persist loc 0/rsp1/cpu0

13. Manually copy any eem scripts from RSP440 to RSP880-LT disks if applicable

The RSP880-LT is now ready to claim mastership from the RSP440 so that the second RSP880-LT module can be installed into the router. You now need to do a “redundancy switchover” between the RSP’s. Once the RSP880-LT is in the Active state, you can remove the RSP440 from Slot1 and insert the second RSP880-LT in its place.

14. Perform a redundancy switchover between the Active RSP440 and Standby RSP880-LT:

RP/0/RSP1/CPU0:ASR9K-G-1102#redundancy sw

Mon Feb 3 10:17:52.088 UTC

Proceed with switchover 0/RSP1/CPU0 -> 0/RSP0/CPU0? [confirm]

Initiating switch-over.

RP/0/RSP0/CPU0:Feb 3 10:17:52.981 UTC: rmf_svr[399]: %HA-REDCON

Reboot on ASR9006-V2 RSP3 (0x100306) in slot 1

By rmf_svr via REBOOT_CAUSE_FAILOVER (4000004)

Current time: 2020-02-03 10:17:53.077, Up time: 38m 59s

Release mastership on RSP3

Using UDMA5

Normal reboot

After the switchover, remove RSP440 from Slot 1 and insert the second RSP880-LT into Slot 1:

Once the second RSP880-LT has been installed into Slot 1, the module will boot up and start synchronising all the configuration and NSR state information from the Active RSP. After this process is complete, the second RSP880-LT will transition in the IOS XR RUN state – in the Standby mode.

NOTE: In case newly inserted second RSP880-LT fails to boot up, park standby RSP880-LT in Rommon. Type “set” to list Rommon variables. Check for the presence of Rommon variable “RSP_LINK_1G=1”. If set, unset RSP_LINK_1G and reset card by following below steps.

rommon 1 > set

PS1=rommon ! >

?=0

IOX_ADMIN_CONFIG_FILE=

ACTIVE_FCD=1

IPXE_PREF=0

CLUSTER_RACK_ID=0

RSP_LINK_1G=1

CLUSTER_NO_BOOT=

BOOT_DEV_SEQ_CONF=

BOOT_DEV_SEQ_OPER=

RMEMVA=

BSI=0

BOOT=disk0:asr9k-os-mbi-6.4.2/0x100305/mbiasr9k-rsp3.vm,1;

rommon 3 > priv

You now have access to the full set of monitor commands.

Warning: some commands will allow you to destroy your

configuration and/or system images and could render

the machine unbootable.

rommon 4 > unset RSP_LINK_1G

rommon 5 > sync

rommon 6 > sync

rommon 7 > reset -h

Resetting hard .......

!oting Main Processor

Transferring Console

RP/0/RSP0/CPU0:ASR9K-G-1102#sh plat

Mon Feb 3 11:02:58.048 UTC

Node Type State Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0 A9K-RSP880-LT-TR(Active) IOS XR RUN PWR,NSHUT,MON

0/RSP1/CPU0 A9K-RSP880-LT-TR(Standby) IOS XR RUN PWR,NSHUT,MON

0/0/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/0/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/0/1 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/1/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/1/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/1/1 A9K-MPA-20X1GE OK PWR,NSHUT,MON

0/2/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/2/0 A9K-MPA-1X40GE OK PWR,NSHUT,MON

0/2/1 A9K-MPA-1X40GE OK PWR,NSHUT,MON

Once the RSP’s have converged and synchronized all the NSR state information between them, you need to run Steps 11, 12, and 13 again. All the redundancy groups should be in the Ready state and the Backup RSP should be NSR-ready.

FPD Upgrade Procedure

As part of this migration process it should not be necessary to upgrade any FPD (Field Programmable Devices). However, please issue the “show hw-module fpd location all” command to verify if the RSP’s need to be upgraded. If required, the RSP880-LT modules can be upgraded using the following commands:

#admin upgrade hw-module fpd all location 0/rsp0/cpu0

#admin upgrade hw-module fpd all location 0/rsp1/cpu0

Post-Upgrade Steps

Since the RSP880-LT modules use a 10Gig EOBC link between them, we need to remove the ROMMON variable that was previously configured. This can be done from the command line.

Remove the ROMMON variable for the EOBC:

RP/0/RSP0/CPU0:ASR9K-G-1102#run nvram_rommonvar RSP_LINK_1G 0

Mon Feb 3 10:48:07.488 UTC

Set RSP_LINK_1G=0

RP/0/RSP0/CPU0:ASR9K-G-1102#run nvram_rommonvar ALDRIN_VERSION_DISABLE 0

Mon Feb 3 10:48:23.040 UTC

Set ALDRIN_VERSION_DISABLE=0

RP/0/RSP0/CPU0:ASR9K-G-1102#run nvram_rommonvar RSP4L_ALDRIN_1G 0

Mon Feb 3 10:48:36.360 UTC

Set RSP4L_ALDRIN_1G=0

RP/0/RSP0/CPU0:ASR9K-G-1102#

Verify that the ROMMON variable for 1Gig EOBC has been disabled:

RP/0/RSP0/CPU0:ASR9K-G-1102#more nvram:/classic-rommon-var

Mon Feb 3 10:48:41.460 UTC

PS1 = rommon ! > , ACTIVE_FCD = 1, IP_ADDRESS = 172.18.31.1, IP_SUBNET_MASK = 255.248.0.0, DEFAULT_GATEWAY = 172.16.255.254, ? = 0, IPXE_PREF = 0, CLUSTER_RACK_ID = 0, TURBOBOOT = , BSI = 0, BOOT = disk0:asr9k-os-mbi-6.4.2/0x100305/mbiasr9k-rsp3.vm,1;, RMEMVA = , CLUSTER_NO_BOOT = , BOOT_DEV_SEQ_CONF = , BOOT_DEV_SEQ_OPER = , RSP_LINK_1G = 0, ALDRIN_VERSION_DISABLE = 0, RSP4L_ALDRIN_1G = 0, confreg = 0x102RP/0/RSP0/CPU0:ASR9K-G-1102#

RP/0/RSP0/CPU0:ASR9K-G-1102#

Verify that the RSP880-LT EOBC ports are running at 10Gig with the following commands. The output should say 10G/KR/1-lane for all the commands:

show controllers epm-switch port-status 8 loc 0/rsp0/cpu0 | in Mode

show controllers epm-switch port-status 9 loc 0/rsp0/cpu0 | in Mode

show controllers epm-switch port-status 8 loc 0/rsp1/cpu0 | in Mode

show controllers epm-switch port-status 9 loc 0/rsp1/cpu0 | in Mode

Note: If there is no output from previous command, it means that the port mapping is not clear. In this case, follow below steps.

show controllers epm-switch port-mapping location 0/RSP0/CPU0 << find the port connected to peer RP and then use command:

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-mapping locatio$

Tue Feb 4 16:32:11.884 UTC

Port | Link Status | Vlan | Connected to

------------|-----------------|---------------|---------------

4 | Up | VLAN_EOBC_0 | LC_EOBC_0_0

5 | Up | VLAN_EOBC_0 | LC_EOBC_1_0

6 | Up | VLAN_EOBC_0 | LC_EOBC_2_0

7 | Down | VLAN_EOBC_0 | LC_EOBC_3_0

8 | Up | VLAN_EOBC_0 | PEER_RP_0_0

9 | Up | VLAN_EOBC_1 | PEER_RP_0_1

10 | Up | VLAN_PUNT | TORNADO_PL_0_0

11 | Up | VLAN_PUNT | TORNADO_PL_1_0

12 | Up | VLAN_PUNT | TORNADO_PL_2_0

13 | Up | VLAN_PUNT | TORNADO_PL_3_0

14 | Down | VLAN_PUNT | TORNADO_PL_4_0

15 | Down | VLAN_PUNT | TORNADO_PL_5_0

16 | Down | VLAN_PUNT | TORNADO_PL_6_0

17 | Down | VLAN_PUNT | TORNADO_PL_7_0

23 | Up | VLAN_PUNT | NIANTIC_0_0

24 | Up | VLAN_10XGE_SWITCH | NIANTIC_0_1

25 | Up | VLAN_10XGE_SWITCH | 10XGE SWITCH_0_0

28 | Up | VLAN_EOBC_1 | NIANTIC_0_0

29 | Up | VLAN_EOBC_0 | NIANTIC_0_1

30 | Down | VLAN_EOBC_0 | OFFLOAD_0_0

31 | Down | VLAN_EOBC_0 | OFFLOAD_1_0

----------------------------------------------------------

show controllers epm-switch port-status <port> loc 0/rsp0/cpu0 << where <port> is from previous output related to PEER_RP)

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-status 8 loc 0/rs$

Tue Feb 4 16:37:09.369 UTC

Mode : 10G/KR/1-lane

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-status 9 loc 0/rs$

Tue Feb 4 16:37:09.775 UTC

Mode : 10G/KR/1-lane

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-status 8 loc 0/rs$

Tue Feb 4 16:37:09.827 UTC

Mode : 10G/KR/1-lane

RP/0/RSP0/CPU0:ASR9K-G-1102#show controllers epm-switch port-status 9 loc 0/rs$

Tue Feb 4 16:37:09.881 UTC

Mode : 10G/KR/1-lane

At this point the hardware upgrade has been completed. Please do the necessary checks and verifications to confirm that the router is fully operational and traffic patterns are as expected

Hardware Upgrade from RSP440 to RSP880-LT

Prerequisites

Upgrade Process

FPD Upgrade Procedure

Post-Upgrade Steps