ASR9000 64-bit IOS XR: Increase RAM available to XR VM by disabling ISSU

Aleksandar Vidakovic · ‎04-26-2019

ASR9000 routers support for In Service Software Upgrade for Cisco IOS XR 64 Bit. ISSU significantly minimises disruption on the control plane and forwarding plane during software upgrade.

To allow for a new instance of the XR operating system, running the target XR release, to be created while the base XR release is still active, a second XR VM has to be spawned on all line cards. While on 32-bit IOS XR the total physical RAM is available to LC CPU, on 64-bit IOS XR the total physical RAM must be distributed between 4 entities:

host OS
Admin VM
XR VM
XR VM v2

All flavours of Tomahawk line cards have 24GB physical RAM available to LC CPU. On 64-bit IOS XR, the XR VM has access to 10GB.

As 32-bit IOS XR runs directly on hardware, you can see the size of physical RAM by running the "show memory summary" command:

RP/0/RSP1/CPU0:ariad#sh platform | i "Node|100GE-TR"
Node            Type                      State            Config State
0/3/CPU0        A9K-8X100GE-TR            IOS XR RUN       PWR,NSHUT,MON

RP/0/RSP1/CPU0:ariad#

RP/0/RSP1/CPU0:ariad#sh memory summary location 0/3/CPU0

node:      node0_3_CPU0
------------------------------------------------------------------
Physical Memory: 24576M total
 Application Memory : 24253M (18554M available)
 Image: 82M (bootram: 82M)
 Reserved: 224M, IOMem: 0, flashfsys: 0
 Total shared window: 431M
RP/0/RSP1/CPU0:ariad#

On 64-bit IOS XR the same command shows the amount of RAM available to XR VM on LC CPU:

RP/0/RSP0/CPU0:ios#sh platform | i "Node|GE-TR"
Node              Type                       State             Config state
0/0/CPU0          A9K-8X100GE-TR             IOS XR RUN        NSHUT
RP/0/RSP0/CPU0:ios#sh memory summ loc 0/0/cPU0
Wed Apr  3 08:02:20.007 UTC

node:      node0_0_CPU0
------------------------------------------------------------------

 Physical Memory: 10691M total (4727M available)
 Application Memory : 10691M (4727M available)
 Image: 4M (bootram: 0M)
 Reserved: 0M, IOMem: 0M, flashfsys: 0M
 Total shared window: 300M

RP/0/RSP0/CPU0:ios#

In some deployment scenarios XR VM on LC CPU may require more RAM, e.g.: combination of full Internet table and large netflow cache. If your topology doesn't support ISSU or if you simply need more RAM and do not care about ISSU, you can double the amount of RAM available to XR VM by disabling ISSU.

ISSU is disabled globally, affecting all line cards. You cannot selectively enable/disable ISSU on individual line cards. Command for disabling ISSU is available in the Admin VM Configuration mode:

RP/0/RSP0/CPU0:ios#admin   
sysadmin-vm:0_RSP0# config
sysadmin-vm:0_RSP0(config)# sdr default-sdr issu disable
sysadmin-vm:0_RSP0(config)# commit
The following warnings were generated:
  '': SDR issu-flag is being changed. Affected SDR might reload.
Proceed? [yes,no] yes
Commit complete.
sysadmin-vm:0_RSP0(config)#

At this point all LCs are reloaded. After the XR VM on line cards boots up, it you should see almost a double amount of RAM available.

Caution:

CARE must be taken when you want to re-enable ISSU
- If you simply configure 'no sdr default-sdr' followed by a commit, then all XR VMs will disappear from every node except the Active RSP. Recovery is laborious and time consuming.
If there is a mix of Lightspeed and Tomahawk line card in the chassis, you need a SMU for CSCvk63470.

AndresSanchez2854 · ‎07-02-2019

Hi Aleksandar, great post.

We are starting to test in Lab the upgrade of a ASR 6.4.2 32bit to 6.4.2 64bits.

Before we start we want to have a collection of database of several resources, like the memory on the line cards and the CPU to have a good idea of how the conversion to 64 will impact the production devices. So we start polling via SNMP several data.

We are a little confused about the purpose of each memory of linecard and RSP:

Physical Memory is the RAM memory of the linecard

Application Memory is the Memory assigned where the programs run.

Image is the size of the memory occupied by the XR os.

We aren't sure about the Reserved Memory.

When we poll we also have the memory available for Processor, Reserved and Image, where Reserved and Image are always zero. This should be the same in a 64bit XR? There will be a new entity to start polling on each linecard? (example: LC 0/0/CPU0 will have the memory of Processor, Reserved and Image for XR VM and another set of memory Proc, Reserved, and Image for the XR VM2)

Thanks for your work and time, deeply appreciated.

Andres.-

Aleksandar Vidakovic · ‎07-03-2019

hi Andres,

thank you for the feedback on the post.

Inside XR VM, the "Physical Memory" and "Application Memory" are the same. They show the amount of memory allocated to the XR VM. Part of this memory is taken by the IOS XR binary itself. The "Image" and "Reserved" you can ignore for all practical purposes.

I'm sharing below the method to confirm the physical RAM memory available on the board. You have to go to Host OS prompt and run the "cat /proc/meminfo" from there.

RP/0/RSP0/CPU0:9904-E_eXR#admin

sysadmin-vm:0_RSP0# run

[sysadmin-vm:0_RSP0:~]$chvrf 0 bash

[sysadmin-vm:0_RSP0:~]$ssh my_host

<...output omitted...>

[host:~]$ cat /proc/meminfo

MemTotal: 32764948 kB

MemFree: 830404 kB

MemAvailable: 1203612 kB

<...output omitted...>

[host:~]$

Hope this helps,

/Aleksandar

thieu · ‎10-11-2021

Hi Aleksandra,

This post is very helpful!

Currently, I had an issue with high memory consumption on ASR9922 eXR 7.1.3 with A99-RP3-TR. When bring up the router with clean configuration, the used memory is 8GB without any feature activated on the router (Available 12G). My question is : can I use the same command "sdr default-sdr issu disable" to disable ISSU on RP to get more memory for Router?

Thanks

Hieu Trinh

Aleksandar Vidakovic · ‎10-11-2021

@thieu

hi Trinh,

I'm glad to see that you've found the post helpful.

On RP we don't reserve RAM for ISSU because during the ISSU the standby RP is upgraded first and then we do the RP failover. "Local" failover to another XR VM is therefore required only on line cards.

/Aleks