Migrating from RSP440 to RSP880 on ASR9000

Eddie Chami · ‎01-18-2016

Migrating from RSP440 to RSP880 on ASR9000

1) Prerequisites:

Console access to the router and Router must be running release 5.3.2 or (5.3.1 with engineering SMU)

System must not have 800G (tomahawk) line cards, (see caveat (a) in list below)

RSP1 is the active RSP440 (see caveat(b) in list below)

This process only works when upgrading RSP-440-TR/SE to RSP-880. This process does NOT apply to A9K-RSP-4G or A9K-RSP-8G(AKA RSP2), an upgrade from RSP2 to RSP880 is not graceful.

Before the upgrade process starts, and prior to removing any RSP from the chassis, make sure that “show redundancy” shows NSR and standby card is ready.

Make sure any none 5.3.x software is removed from the system (ie install remove *4.3.4*)

Enable console logging on the router and on the terminal application (optional)

2) Caveats:

The system must NOT have Tomahawk (400G or 800G) Line cards in the chassis prior to attempting this as the EOBC link between Tomahawk and newly inserted RSP880 will be down which will interfere with the failover from RSP440 to RSP880. If there are Tomahawk cards in the chassis, they need to be removed or powered down for the upgrade to be successful. ‘admin-config #hw-module power disable location 0/1/cpu0’

While doing RSP440 to RSP880 upgrade, Active RSP440 must be in slot 1, else the user will observe online diag fabric punt failure to LC on the newly standby RSP880.

Remove Trident Linecards from the system as they are incompatible with the RSP880.

3) Upgrade process:

RSP1 is the active RSP440 running 5.3.2 or 5.3.1 + SMU

If RSP0 slot is active and RSP1 is standby then first OIR/Remove RSP0 slot card (or do redundancy switchover from RSP0 to RSP1) to make RSP1 as active and then insert RSP880 in slot0.

Issue show redundancy and make sure NSR and Standby card is ready

RP/0/RSP0/CPU0:vkg7RO#show redundancy summary

Wed Jun 17 11:18:51.391 PST

Active/Primary Standby/Backup

-------------- --------------

0/RSP0/CPU0(A) 0/RSP1/CPU0(S) (Node Ready, NSR: Ready)

0/RSP0/CPU0(P) 0/RSP1/CPU0(B) (Proc Group Ready, NSR: Ready)

RP/0/RSP0/CPU0:vkg7RO#

RP/0/RSP0/CPU0:vkg7RO#show redundancy

Wed Jun 17 11:20:19.086 PST

Redundancy information for node 0/RSP0/CPU0:

==========================================

Node 0/RSP0/CPU0 is in ACTIVE role

Node Redundancy Partner (0/RSP1/CPU0) is in STANDBY role

Standby node in 0/RSP1/CPU0 is ready

Standby node in 0/RSP1/CPU0 is NSR-ready

Node 0/RSP0/CPU0 is in process group PRIMARY role

Process Redundancy Partner (0/RSP1/CPU0) is in BACKUP role

Backup node in 0/RSP1/CPU0 is ready

Backup node in 0/RSP1/CPU0 is NSR-ready

Group Primary Backup Status

--------- --------- --------- ---------

v6-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

mcast-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

netmgmt 0/RSP0/CPU0 0/RSP1/CPU0 Ready

v4-routing 0/RSP0/CPU0 0/RSP1/CPU0 Ready

central-services 0/RSP0/CPU0 0/RSP1/CPU0 Ready

dlrsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

dsc 0/RSP0/CPU0 0/RSP1/CPU0 Ready

Reload and boot info

----------------------

A9K-RSP440-SE reloaded Fri Jun 12 14:40:22 2015: 4 days, 20 hours, 39 minutes ago

Active node booted Fri Jun 12 20:52:24 2015: 4 days, 14 hours, 27 minutes ago

Last switch-over Mon Jun 15 10:43:34 2015: 2 days, 36 minutes ago

Standby node boot Mon Jun 15 10:44:53 2015: 2 days, 35 minutes ago

Standby node last went not ready Mon Jun 15 10:46:27 2015: 2 days, 33 minutes ago

Standby node last went ready Mon Jun 15 10:47:27 2015: 2 days, 32 minutes ago

There have been 3 switch-overs since reload

Active node reload "Cause: Initiating switch-over."

Standby node reload "Cause: Initiating switch-over."

RP/0/RSP0/CPU0:vkg7RO#

3) Remove the standby RSP440 (RSP0)

4) Insert RSP880 (A) in slot 0.

5) Issue Ctrl-c to break into ROMMON of inserted RSP880

(If this is missed, the RSP880 will go into a boot cycle loop, and there will be another chance to break into rommon on the next bootup, no side effects apart from syslog msgs on the active RSP).

NOTE: if your running 5.3.x and greater releases, you may get a menu to choose various boot options, you will need to select "boot IOS XR in 32-bit mode", if the requirement is to stat on 32 bit XR.

6) From ROMMON prompt on RSP880 (A), set ROMMON variable to force 1GE Mode for Peer RSP communication (RSP_LINK_1G=1), and check config register is set correctly in ROMMON

rommon B1> RSP_LINK_1G=1

rommon B1> sync

7) Reset RSP880 (A) card, it will boot up to standby state which will synchronize configuration from Active RSP440 in Slot 1

rommon B1> reset -h

8) Verify the RSPs have reached full synchronization with “show redundancy” command and the groups are in “ready” state.

show redundancy

9) Verify the RSPs have synchronized the SNMP engine ID and SNMP ifindex-table

#more disk0:snmp/ifindex-table loc 0/rsp0/cpu0

#more disk0:snmp/ifindex-table loc 0/rsp1/cpu0

#more disk0:snmp/snmp_persist loc 0/rsp0/cpu0

#more disk0:snmp/snmp_persist loc 0/rsp1/cpu0

manually copy any eem scripts from RSP440 to RSP880 disks if applicable

10) Failover from active RSP440 (slot 1) to standby RSP880 in (Slot0)

“redundancy switchover”

NOTE: A CLI switchover is needed, a physical OIR of the RP will not do the trick.

11) Verify the RSP880 becomes active RSP and configuration is present

At this point the RSP440 must be removed/eject from slot 1.

12) Insert RSP880 (B) and allow it to boot to standby. Do *not* set the ROMMON variable

13) Issue show redundancy to confirm the RSPs have sync’d up and in correct state

'show platform’

'show redundancy'

manually copy any eem scripts from Active RSP to Standby disks if applicable

14) On Active RSP880 Clear ROMMON variable from XR

run nvram_rommonvar RSP_LINK_1G 0

more nvram:/classic-rommon-var

In case newly inserted second RSP880 fails to boot up, park standby RSP880 in Rommon.

Type “set” to list Rommon variables. Check for the presence of Rommon variable “RSP_LINK_1G=1”.

If set, unset RSP_LINK_1G and reset card by following below steps.

rommon B1> priv

rommon B1> unset RSP_LINK_1G

rommon B1> sync

rommon B1> reset -h

15) Perform FPD upgrade on the RSP880 (optional)

admin upgrade hw-module fpd all location 0/rsp0/cpu0

admin upgrade hw-module fpd all location 0/rsp1/cpu0

16) Deactivate and remove the SMU.

17) Check the EOBC ports are running at 10G with the following four commands, the output should be “10G/KR/1-lane”

show controllers epm-switch port-status 54 loc 0/rsp0/cpu0 | in Mode

show controllers epm-switch port-status 55 loc 0/rsp0/cpu0 | in Mode

show controllers epm-switch port-status 54 loc 0/rsp1/cpu0 | in Mode

show controllers epm-switch port-status 55 loc 0/rsp1/cpu0 | in Mode

18) Final checks to ensure system is optional and functional

Note; this operation has been tested to be graceful with NSR configured

danjlopez · ‎04-11-2016

Anyone have anything similar to this for a RSP4G to RSP440 migration?

Thanks,

Dan

Eddie Chami · ‎04-11-2016

Dan,

Its a different process since RSP4G and RSP440 can't co-exist at the same time.

You'll have to backup the config of the RSP4G offline, insert the RSP440's. Copy the config back, its much simpler, but more intrusive.

Regards

Eddie.

danjlopez · ‎04-12-2016

That is what I was expecting, but could not find supporting documentation. Thanks for confirming Eddie.

Regards,

Dan

iptechttk · ‎08-23-2016

Hello,

Does it has the difference if we started migration when 0/RP0/CPU0 in ACTIVE role?

CDWgpowers · ‎10-19-2016

We just performed this upgrade as well and only ran into one variance during the procedure. At step 5) you will need to select the option to boot IOS XR in 32-bit mode before the rommon prompt will be presented. This may come without saying for those very familiar with the ASR9K/XR architecture but maybe not for others so I thought I would include it.

Eddie Chami · ‎10-19-2016

Hi, i've updated section 5, i assume your running 5.3.x or greater.. right?

CDWgpowers · ‎10-19-2016

Correct. 5.3.3

iptechttk · ‎10-19-2016

Yes. We have use 5.3.3 release.

takao.nakamaru · ‎10-25-2016

I realize that perhaps we are asking for a lot here, but have you tested the migration from RSP440 => RSP880 with BNG functionality enabled (and active PPPoE sessions_?

What happens to pre-existing ppp sessions when in step 10 you perform the Redundancy Switchover from RSP440 => RSP880?

(Can the PPP sessions - and the relevant structures - also be synchronized between RSP440 & RSP880 or they must be re-established after the switchover)?

Aleksandar Vidakovic · ‎10-25-2016

If you wait long enough for sessions to be synced between active and standby, the RSP failover should be graceful. Please note before XR release 5.3.4 we were not testing BNG on RSP880. In 5.3.4 we have tested RSP880 in limited deployment scenarios. Which release are you running?

regards,

/Aleksandar

takao.nakamaru · ‎10-25-2016

Hello Aleksandar

We do not currently use 5.3.4, but by the time we shall perform this migration, we will have a 6.x.x variant installed in the RSP440

Regards

Aleksandar Vidakovic · ‎10-26-2016

In that case this should work. Use the command "sh subscriber infra readiness" to confirm that the subscriber info was fully replicated on the newly inserted RSP880 before you initiate the RP failover. Please make sure to validate this in the lab within your specific BNG deployment scenario before attempting in production.

Carlos A. Silva · ‎11-30-2016

Hi,

When following this process, what happens if you have previous versions (5.1.x) installed but not active in RSP1 (RSP440) in the first few steps? Should it work without a problem?

I'm currently researching a customer that had problems, apparently because of this.

Thanks,

c.

This is an extract of what customer sees (different versions when syncing):

Nov 27 10:54:42.487: Install Setup: Complete

Nov 27 10:54:42.487: Install Setup: Syncing package 'disk0/asr9K-doc-supp-5.3.2'

:

Nov 27 10:54:44.203: Install Setup: Complete

Nov 27 10:54:44.204: Install Setup: Syncing package 'disk0/asr9k-ce-5.1.2':

Nov 27 10:54:44.851: Install Setup: Complete

Nov 27 10:54:44.852: Install Setup: Syncing package 'disk0/asr9k-bng-5.3.2':

Nov 27 10:54:45.254: Install Setup: Complete

Nov 27 10:54:45.254: Install Setup: Syncing package 'disk0/iosxr-bng-5.3.2':

Nov 27 10:54:48.455: Install Setup: Complete

Nov 27 10:54:48.457: Install Setup: Syncing package 'disk0/asr9k-fwding-5.1.2':

Nov 27 10:55:01.462: Install Setup: Complete

Nov 27 10:55:01.469: Install Setup: Syncing package 'disk0/asr9k-9000v-nV-supp-5

.3.2':

Nov 27 10:55:04.245: Install Setup: Complete

Nov 27 10:55:04.246: Install Setup: Syncing package 'disk0/asr9k-base-5.1.2':

Nov 27 10:57:03.118: Install Setup: Failed: Unable to sync all of the contents

of '/disk0/asr9k-base-5.1.2'

Nov 27 10:57:03.134: Install Setup: Failed to sync some packages or meta-data

Nov 27 10:57:03.134: Install Setup: Install Setup (pid 876588) has failed to pre

pare this node successfully and will now exit: (2949233664) 'Subsystem(8083)' de

tected the 'fatal' condition 'Code(30)'

Torben Joergensen · ‎01-05-2017

Hi

I just tried this in our lab and failed. What i did´nt get right was the command: reset -h

I only issued: reset

Everything looked good - but when i made the: admin redundancy switchover - i got all kinds of spooky error messages. The router stopped forwarding packets. Had to power off and power on to get it up and running.

It came back to life.. weee :)

But - my question.. could it be that i forgot the -h in the reset command?

Don´t know what -h does. Any idea´s.?

Regards
Torben

Eddie Chami · ‎01-05-2017

Yes you need to follow the exact instructions, also you want to make sure your not running into CSCva30444. Which BIOS/XR version is running on the RSP880 or RP2?

Eddie.