WS-C3750X-48P HRPC qos request - High CPU Utilization

[Fulvio Ferreira]

I have a 4 Cisco 3750X stacked showing a high CPU utilization and I would like help to find the root cause:

I found the HRPC qos request process with high utilization.

CPU utilization for five seconds: 95%/1%; one minute: 92%; five minutes: 81%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
206 235170 8011 29355 48.00% 46.21% 35.34% 0 HRPC qos request
189 92625 11696 7919 13.48% 14.47% 12.71% 0 Hulc LED Process
88 37263 6400 5822 4.17% 4.82% 4.17% 0 RedEarth Tx Mana
87 26185 10655 2457 3.04% 3.35% 3.09% 0 RedEarth I2C dri
488 3616 10433 346 2.72% 0.68% 0.47% 0 PIM Process
4 15308 628 24375 2.40% 2.10% 1.95% 0 Check heaps
191 3017 1068 2824 1.60% 0.27% 0.30% 0 hl3mm
135 9169 620 14788 1.60% 1.71% 1.40% 0 hpm counter proc

IOS Version:15.2(4)E6 

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 54 WS-C3750X-48P 15.2(4)E6 C3750E-UNIVERSALK9-M
2 54 WS-C3750X-48P 15.2(4)E6 C3750E-UNIVERSALK9-M
3 54 WS-C3750X-48P 15.2(4)E6 C3750E-UNIVERSALK9-M
4 54 WS-C3750X-48P 15.2(4)E6 C3750E-UNIVERSALK9-M

Thanks

[Leo Laohoo]

---

@Fulvio Ferreira wrote:

HRPC qos request

---

Turn off QoS.

[Fulvio Ferreira]

I tried to remove qos... without success.

I will open a TAC for this issue.

Thanks

[Georg Pauwen]

Hello,

on a side note, check if the bug (and recommended workaround) might apply:

3750 High CPU in HRPC qos request
CSCsm41883
Description
Symptom

In some instances, a 3750 family switch may experience high cpu (>90%) in the "HRPC qos request" process when a new client device is connected to the switch for the first time.

Conditions:

- new connection
- QoS service-policy applied to the interface

Workaround:

- CPU should remain stable after initial spike. Issue is not seen if the same device reconnects back to the same interface.

- A potential work-around to further reduce CPU Processing will be as follows,

(1) For ports connecting to Cisco-phone, use
"auto qos voip cisco-phone"

(2) For ports NOT connecting to Cisco-phone,

(2a) remove "mls qos trust device cisco-phone" (since this line won't work for non-Cisco Phone anyway), If trust is needed,
a separte class-map can be configured as follows to have the same behavior.

(2b) Define a different policy-map for each interface, using the same class-maps.

-----------------------------------------------

ip access-list extended ip-any
permit ip any any

class-map match-all trust-dscp
match access-group name ip-any

policy-map Edge_QOS_Marking-1

class VOICE

set ip dscp ef

police 128000 8000 exceed-action drop

class CALL-SIGNALLING

set ip dscp cs3

police 32000 8000 exceed-action policed-dscp-transmit
...
class trust-dscp
trust dscp

...

policy-map Edge_QOS_Marking-50

class VOICE

set ip dscp ef

police 128000 8000 exceed-action drop

class CALL-SIGNALLING

set ip dscp cs3

police 32000 8000 exceed-action policed-dscp-transmit
...
class trust-dscp
trust dscp
...

interface g1/0/1
service-policy input Edge_QOS_Marking-1

interface g1/0/2
! If connected to cisco-phone
auto qos voip cisco-phone
...

interface g1/0/50
service-policy input Edge_QOS_Marking-50

[Fulvio Ferreira · Accepted solution]

Hi.. Follow the solution according CISCO TAC:

Problem Analysis

PID | Runtime  | Invoked | uSecs  | 5 Sec  | 1 Min   | 5 Min   | TTY | Process         

|   ----|----------|---------|--------|--------|---------|---------|-----|------------------

|   07  |  1387785 |  44268  |  31349 | 31.04% |  38.93% |  43.97% |  0  | HRPC qos request    >>>>>

|   89  |  25526   |  2333   |  10941 | 18.24% |  9.88%  |  4.93%  |  2  | SSH Process     

|   90  |  536742  |  62298  |  8615  | 16.32% |  15.52% |  14.82% |  0  | Hulc LED Process

|   88  |  225845  |  36715  |  6151  | 7.36%  |  5.66%  |  5.74%  |  0  | RedEarth Tx Mana

|   87  |  145830  |  63226  |  2306  | 5.28%  |  4.30%  |  4.42%  |  0  | RedEarth I2C dri

This process is used for communication between stacked switches when programming qos entries and therefore if we have a link constantly flapping this would cause the "HRPC qos request" process to be invoked frequently leading to high cpu

--Shutting down gi4/0/5 resolved the issue

Dec  8 05:55:55.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up

Dec  8 05:55:56.933: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/5, changed state to up

Dec  8 05:55:59.189: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to down

Dec  8 05:55:59.550: %LINK-5-CHANGED: Interface FastEthernet0, changed state to administratively down

Dec  8 05:56:00.145: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/5, changed state to down

Dec  8 05:56:03.316: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up

Dec  8 05:57:53.098: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 94%/9%, Top 3 processes(Pid/Util):  383/24%, 214/20%, 190/20%

Dec  8 05:58:18.004: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to down

Dec  8 05:58:20.126: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up

As a work around could you please try to shut the interface gi4/0/5 and check the improvement in CPU.

-  Gi4/0/5 was responsible for the high cpu utilization we were seeing due to the "HRPC qos request" process. This process is used for communication between stacked switches when programming qos entries and therefore if we have a link constantly flapping this would cause the "HRPC qos request" process to be invoked frequently leading to high cpu

--Shutting down gi4/0/5 resolved the issue