IOS-XR: Relationship between Multi-core CPU and CPU utilization

CscTsWebDocs · ‎10-06-2014

If CPU utilization rises due to a performance limit or software failures on IOS-XR, the CPU utilization may continue to keep around 25% or 50%.

This phenomenon occurs because IOS-XR supports multi-core CPUs, and many modules use a multi-core CPU.
* Even with a multi-core CPU, the CPU utilization may rise up to 100% (high CPU) depending on the situation.

How to check CPU Utilization

Command execution example

Pattern 1 (no CPU load)

RP/0/RP0/CPU0:XR#show processes cpu | inc util
CPU utilization for one minute: 1%; five minutes: 1%; fifteen minutes: 1%

Pattern 2 (100% CPU load)

RP/0/RP0/CPU0:XR#show processes cpu | inc util
CPU utilization for one minute: 100%; five minutes: 76%; fifteen minutes: 26%

Pattern 3 (50% CPU load)

RP/0/RP0/CPU0:XR#show processes cpu | inc util
CPU utilization for one minute: 50%; five minutes: 50%; fifteen minutes: 34%

Pattern 4 (25% CPU load)

RP/0/RP0/CPU0:XR#show processes cpu | inc util
CPU utilization for one minute: 25%; five minutes: 25%; fifteen minutes: 22%

Each process of IOS-XR contains a thread, which is running on another CPU core.
In the example shown below, the status of each BGP process running on four-core CPU RPs are verified by using the "show processes" command.From the TID value, we can see that 23 threads exist in the BGP processes.From the CPU value, we can see that each thread is running by being separated into four cores, from 0 to 3.

Detailed Information on BGP Processes

Command execution example

RP/0/RSP0/CPU0:ASR9K#show processes bgp detail
                  Job ID: 1049 << Used to confirm the thread name
                     PID: 573758
         Executable path: /disk0/iosxr-routing-4.3.1/0x100000/bin/bgp
<snip>

JID   TID CPU Stack pri state        TimeInState    HR:MM:SS:MSEC   NAME
1049   1    1 392K 10 Receive        0:00:12:0222    0:00:00:0347 bgp
1049   2    0 392K 10 Receive      359:12:28:0410    0:00:00:0000 bgp
1049   3    2 392K 10 Receive      359:12:28:0392    0:00:00:0000 bgp
1049   4    3 392K 10 Receive        0:00:03:0906    0:00:00:0017 bgp
1049   5    1 392K 10 Receive      359:12:18:0812    0:00:00:0000 bgp
1049   6    2 392K 10 Sigwaitinfo 359:12:19:0701    0:00:00:0000 bgp
1049   7    3 392K 10 Receive        0:01:49:0982    0:00:00:0002 bgp
1049   8    2 392K 10 Receive        0:00:58:0570    0:00:00:0607 bgp
1049   9    1 392K 10 Receive        0:00:49:0977    0:00:00:0006 bgp
1049   10   3 392K 10 Nanosleep      0:00:00:0567    0:00:00:0044 bgp
1049   11   3 392K 10 Receive        0:00:49:0977    0:00:00:0011 bgp
1049   12   3 392K 10 Receive        0:00:02:0636    0:00:43:0721 bgp
1049   13   3 392K 10 Receive        0:00:00:0600    0:00:44:0163 bgp
1049   14   2 392K 10 Receive        0:00:58:0557    0:00:00:0012 bgp
1049   15   1 392K 10 Receive        0:00:58:0564    0:00:00:0007 bgp
1049   16   3 392K 10 Receive        0:00:50:0013    0:00:00:0003 bgp
1049   17   1 392K 10 Receive        0:00:50:0004    0:00:00:0001 bgp
1049   18   3 392K 10 Receive        0:00:58:0120    0:00:00:0010 bgp
1049   19   3 392K 10 Receive        0:00:08:0833    0:00:21:0598 bgp
1049   20   2 392K 10 Receive      359:11:28:0520    0:00:00:0000 bgp
1049   21   2 392K 10 Receive        0:00:00:0805    0:00:00:0006 bgp
1049   22   1 392K 10 Receive        0:00:49:0978    0:00:00:0000 bgp
1049   23   2 392K 10 Receive      359:09:55:0658    0:00:00:0005 bgp

Name of each thread in the BGP processes can be confirmed by "show processes threadname <jid>".
* Specify Job ID confirmed by "show processes" for jid.

Name of Each Thread in BGP Processes

Command execution example

RP/0/RSP0/CPU0:ASR9K#show processes bgp threadname 1049 << Specify jid confirmed by show processes bgp

JID    TID ThreadName      pri state      TimeInState     NAME
1049   1    bgp-io-control 10 Receive        0:00:06:0598 bgp
1049   2    cdm_evm_thread 10 Receive      359:21:47:0206 bgp
1049   3                    10 Receive      359:21:47:0188 bgp
1049   4    bgp-rpki        10 Receive        0:00:02:0590 bgp
1049   5    spp_nsr_client_conn_monitor 10 Receive      359:21:37:0608 bgp
1049   6                    10 Sigwaitinfo 359:21:38:0497 bgp
1049   7    bgp-label       10 Receive        0:01:08:0769 bgp
1049   8    bgp-mgmt        10 Receive        0:00:17:0357 bgp
1049   9    bgp-rib-upd-0   10 Receive        0:00:08:0764 bgp
1049   10   lspv_lib BGPv4 10 Nanosleep      0:00:04:0253 bgp
1049   11   bgp-rib-upd-1   10 Receive        0:00:08:0764 bgp
1049   12   bgp-io-read     10 Receive        0:00:01:0332 bgp
1049   13   bgp-io-write    10 Receive        0:00:01:0332 bgp
1049   14   bgp-router      10 Receive        0:00:17:0344 bgp
1049   15   bgp-import      10 Receive        0:00:17:0351 bgp
1049   16   bgp-upd-gen     10 Receive        0:00:08:0800 bgp
1049   17   bgp-sync-active 10 Receive        0:00:08:0791 bgp
1049   18   bgp-crit-event 10 Receive        0:00:16:0908 bgp
1049   19   bgp-event       10 Receive        0:00:27:0622 bgp
1049   20   bgp-mib-trap    10 Receive      359:20:47:0318 bgp
1049   21   bgp-io-ka       10 Receive        0:00:07:0483 bgp
1049   22   bgp-l2vpn-thr   10 Receive        0:00:08:0766 bgp
1049   23   async           10 Receive      359:19:14:0456 bgp

For example, bgp-router thread in the BGP process use up 100% of the CPU resources.

Since the bgp-router thread is TID(Thread ID) 14, from "show processes bgp detail", we can see that it is running on CPU core 2.Shown below is the result of verifying the CPU utilization by the "show processes cpu" command in this situation.

CPU Utilization of BGP Processes

Command execution example

RP/0/RSP0/CPU0:ASR9K#show processes cpu

CPU utilization for one minute: 25%; five minutes: 16%; fifteen minutes: 3%

PID 1Min 5Min 15Min Process
<snip>
573758 25% 12% 2% bgp
<snip>

Since the CPU utilization is calculated by totaling the four cores, even one core turns out to be running at close to 100% utilization, and the CPU utilization in this environment is about 25%. It is therefore difficult to find out that a multi-core CPU is in a high CPU state, even when monitoring the CPU utilization.
*Although it does not cause problems with the operations, we may have you contact us if your CPU utilization keeps at 25% or 50%.

When you use the "run top_procs -D -i1" command that allows you to check the CPU utilization of each CPU core, you can see that the CPU resources of CPU core 2 are close to 0 (less idling).
* The run command is not supported unless instructed by TAC.

Verifying CPU Utilization for Each CPU Core

Command execution example

RP/0/RSP0/CPU0:ASR9K#run top_procs -D -i1

node0_RSP0_CPU0: 328 procs, 4 cpus, 5.08 delta, 359:54:21 uptime

Memory: 8192 MB total, 6.303 GB free, sample time: Sat Jan 20 08:09 AM:51 2014

cpu 0 idle: 99.30%, cpu 1 idle: 98.74%, cpu 2 idle: 0.56%, cpu 3 idle: 98.60%, total idle: 74.87%, kernel: 0.05%

<snip>

Based on the above, if one process continues to occupy 25% or 50% of CPU utilization when you are using a module with multi-core CPU on IOS-XR, it could be due to a performance limit or software failures, just like in the case of conventional high CPU problems, even when the overall utilization does not stay at 100%. Please feel free to contact TAC if you are concerned about CPU utilization.

The number of CPU cores for RP and LC supported by CRS-1/CRS-3/ASR9000 is described in the following document.

IOS-XR: Number of CPU Cores for RP/LC Supported by CRS-1/CRS-3/ASR9000
https://supportforums.cisco.com/ja/document/12067881

Related Information

Original Document: https://supportforums.cisco.com/ja/document/12067876
Author: Yoichi Kanme
Posted on January 25, 2014