RSP-4G with IOS-XR low free memory

[Rodrigo Rozário]

We recently upgraded some of our ASR9k with RSP-4G from 4.0.3 to 5.3.4. Since that we are experiencing some crashs related to low free memory. Unfortunately we don't have enough flash space to try SP10 in order to see if there is any enhancement with memory utilization. 

Could you guys suggest anything we can do with these routers?

Tue Jul 9 11:40:01.280 BRAZIL
Physical Memory: 4096M total (649M available)
Application Memory : 3897M (649M available)
Image: 69M (bootram: 69M)
Reserved: 128M, IOMem: 2028M, flashfsys: 0
Total shared window: 229M

[Aleksandar Vidakovic]

It would be the best to first monitor the memory utilisation on the router to establish whether there is a memory leak or is there a memory shortage during some transient conditions. Also check which process is consuming most memory and see how can that consumption be reduced. Running 5.3.4 without the latest Service Pack or SMU set equivalent is not something we would recommend. It would very good to upgrade this RSP because the release 5.3.4 has reached End Of SW Maintenance almost a lear ago.

[Aleksandar Vidakovic]

It would be the best to first monitor the memory utilisation on the router to establish whether there is a memory leak or is there a memory shortage during some transient conditions. Also check which process is consuming most memory and see how can that consumption be reduced. Running 5.3.4 without the latest Service Pack or SMU set equivalent is not something we would recommend. It would very good to upgrade this RSP because the release 5.3.4 has reached End Of SW Maintenance almost a lear ago.

[Rodrigo Rozário]

@Aleksandar Vidakovic, thank you for your considerations. In fact we cannot swap this RSP for now and we want to give to this hardware some extra time. Also we have been using 4.0.3 without big issues for years. To be honest, I don't now if there is a better option for the hardware we have here.

To address the memory usage problem, take a look at this output.

Wed Jul 10 14:36:26.357 BRAZIL
JID Text Data Stack Dynamic Dyn-Limit Shm-Tot Phy-Tot Process
------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- -------
1059 1M 6M 484K 431M 1425M 68M 438M bgp
1167 524K 464K 380K 325M 1568M 28M 326M ipv4_rib
334 8K 12K 176K 216M 1024M 29M 217M mibd_interface
60 128K 60K 176K 32M 300M 31M 32M eth_server
363 204K 136K 140K 24M 1568M 10M 24M parser_server
1169 520K 620K 228K 23M 1568M 27M 24M ipv6_rib
1121 564K 736K 56K 21M 300M 13M 22M lspv_server
1200 3M 14M 208K 14M 1568M 48M 28M l2vpn_mgr
58 240K 80K 56K 13M INFINITY 2M 13M dllmgr
1183 3M 1M 196K 13M 1024M 379M 15M pim6
248 1M 1M 120K 13M 800M 127M 15M iedged

So, we mainly have bgp, ipv4_rib, mibd_interface using a big amount of memory. See the available memory right now.

Wed Jul 10 14:40:18.366 BRAZIL
Physical Memory: 4096M total (500M available)
Application Memory : 3897M (500M available)
Image: 69M (bootram: 69M)
Reserved: 128M, IOMem: 2028M, flashfsys: 0
Total shared window: 210M

I think the problem is related to the enabled features in this router, that have at least 2 full routing tables, handreds of interfaces to be queried by snmp every 5 minutes, mpls and some VPNs configured.

Finally, we have flash size restrictions to install the SP10.

[Aleksandar Vidakovic]

hi Rodrigo,

if you decrease the BGP table size on this router, the ipv4_rib size will decrease as well. That should prevent hitting watchdog thresholds. You can also lower the watchdog thresholds using the "watchdog threshold memory minor <n> severe <m> critical <k>" command. 

Regarding disk space, you could remove the FPD pie from the disk once FPD is upgraded. You could also opt for individual SMUs instead of SP.

To optimise the list of SMUs you can still use the CSM Application (https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/smu/csmuser.html) although it's not officially supported any more. A much better option would be to use the CSM Server to optimise the list of SMUs and also orchestrate the installation. CSM Server installs in a docker container. You can download it from https://software.cisco.com/download/home/282414851/type/284777134/release/4.0 

If you're considering a HW upgrade for the RSP, please reach out to your account team to see what would be the optimal choice for you. In general A9K-RSP880-LT should be an interesting option, although that would require an upgrade to 6.2.3 or later and also an upgrade of any Trident line card you have in that chassis to Tomahawk.

/Aleksandar