12-07-2018 03:20 PM - edited 03-08-2019 04:46 PM
I have a 4 Cisco 3750X stacked showing a high CPU utilization and I would like help to find the root cause:
I found the HRPC qos request process with high utilization.
CPU utilization for five seconds: 95%/1%; one minute: 92%; five minutes: 81%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
206 235170 8011 29355 48.00% 46.21% 35.34% 0 HRPC qos request
189 92625 11696 7919 13.48% 14.47% 12.71% 0 Hulc LED Process
88 37263 6400 5822 4.17% 4.82% 4.17% 0 RedEarth Tx Mana
87 26185 10655 2457 3.04% 3.35% 3.09% 0 RedEarth I2C dri
488 3616 10433 346 2.72% 0.68% 0.47% 0 PIM Process
4 15308 628 24375 2.40% 2.10% 1.95% 0 Check heaps
191 3017 1068 2824 1.60% 0.27% 0.30% 0 hl3mm
135 9169 620 14788 1.60% 1.71% 1.40% 0 hpm counter proc
IOS Version:15.2(4)E6
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 54 WS-C3750X-48P 15.2(4)E6 C3750E-UNIVERSALK9-M
2 54 WS-C3750X-48P 15.2(4)E6 C3750E-UNIVERSALK9-M
3 54 WS-C3750X-48P 15.2(4)E6 C3750E-UNIVERSALK9-M
4 54 WS-C3750X-48P 15.2(4)E6 C3750E-UNIVERSALK9-M
Thanks
Solved! Go to Solution.
12-08-2018 05:54 AM
Hi.. Follow the solution according CISCO TAC:
Problem Analysis
PID | Runtime | Invoked | uSecs | 5 Sec | 1 Min | 5 Min | TTY | Process
| ----|----------|---------|--------|--------|---------|---------|-----|------------------
| 07 | 1387785 | 44268 | 31349 | 31.04% | 38.93% | 43.97% | 0 | HRPC qos request >>>>>
| 89 | 25526 | 2333 | 10941 | 18.24% | 9.88% | 4.93% | 2 | SSH Process
| 90 | 536742 | 62298 | 8615 | 16.32% | 15.52% | 14.82% | 0 | Hulc LED Process
| 88 | 225845 | 36715 | 6151 | 7.36% | 5.66% | 5.74% | 0 | RedEarth Tx Mana
| 87 | 145830 | 63226 | 2306 | 5.28% | 4.30% | 4.42% | 0 | RedEarth I2C dri
This process is used for communication between stacked switches when programming qos entries and therefore if we have a link constantly flapping this would cause the "HRPC qos request" process to be invoked frequently leading to high cpu
--Shutting down gi4/0/5 resolved the issue
Dec 8 05:55:55.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up
Dec 8 05:55:56.933: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/5, changed state to up
Dec 8 05:55:59.189: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to down
Dec 8 05:55:59.550: %LINK-5-CHANGED: Interface FastEthernet0, changed state to administratively down
Dec 8 05:56:00.145: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/5, changed state to down
Dec 8 05:56:03.316: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up
Dec 8 05:57:53.098: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 94%/9%, Top 3 processes(Pid/Util): 383/24%, 214/20%, 190/20%
Dec 8 05:58:18.004: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to down
Dec 8 05:58:20.126: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up
As a work around could you please try to shut the interface gi4/0/5 and check the improvement in CPU.
- Gi4/0/5 was responsible for the high cpu utilization we were seeing due to the "HRPC qos request" process. This process is used for communication between stacked switches when programming qos entries and therefore if we have a link constantly flapping this would cause the "HRPC qos request" process to be invoked frequently leading to high cpu
--Shutting down gi4/0/5 resolved the issue
12-07-2018 04:48 PM
@Fulvio Ferreira wrote:
HRPC qos request
Turn off QoS.
12-08-2018 04:00 AM
I tried to remove qos... without success.
I will open a TAC for this issue.
Thanks
12-08-2018 04:39 AM
Hello,
on a side note, check if the bug (and recommended workaround) might apply:
3750 High CPU in HRPC qos request
CSCsm41883
Description
Symptom
In some instances, a 3750 family switch may experience high cpu (>90%) in the "HRPC qos request" process when a new client device is connected to the switch for the first time.
Conditions:
- new connection
- QoS service-policy applied to the interface
Workaround:
- CPU should remain stable after initial spike. Issue is not seen if the same device reconnects back to the same interface.
- A potential work-around to further reduce CPU Processing will be as follows,
(1) For ports connecting to Cisco-phone, use
"auto qos voip cisco-phone"
(2) For ports NOT connecting to Cisco-phone,
(2a) remove "mls qos trust device cisco-phone" (since this line won't work for non-Cisco Phone anyway), If trust is needed,
a separte class-map can be configured as follows to have the same behavior.
(2b) Define a different policy-map for each interface, using the same class-maps.
-----------------------------------------------
ip access-list extended ip-any
permit ip any any
class-map match-all trust-dscp
match access-group name ip-any
policy-map Edge_QOS_Marking-1
class VOICE
set ip dscp ef
police 128000 8000 exceed-action drop
class CALL-SIGNALLING
set ip dscp cs3
police 32000 8000 exceed-action policed-dscp-transmit
...
class trust-dscp
trust dscp
...
policy-map Edge_QOS_Marking-50
class VOICE
set ip dscp ef
police 128000 8000 exceed-action drop
class CALL-SIGNALLING
set ip dscp cs3
police 32000 8000 exceed-action policed-dscp-transmit
...
class trust-dscp
trust dscp
...
interface g1/0/1
service-policy input Edge_QOS_Marking-1
interface g1/0/2
! If connected to cisco-phone
auto qos voip cisco-phone
...
interface g1/0/50
service-policy input Edge_QOS_Marking-50
12-08-2018 05:54 AM
Hi.. Follow the solution according CISCO TAC:
Problem Analysis
PID | Runtime | Invoked | uSecs | 5 Sec | 1 Min | 5 Min | TTY | Process
| ----|----------|---------|--------|--------|---------|---------|-----|------------------
| 07 | 1387785 | 44268 | 31349 | 31.04% | 38.93% | 43.97% | 0 | HRPC qos request >>>>>
| 89 | 25526 | 2333 | 10941 | 18.24% | 9.88% | 4.93% | 2 | SSH Process
| 90 | 536742 | 62298 | 8615 | 16.32% | 15.52% | 14.82% | 0 | Hulc LED Process
| 88 | 225845 | 36715 | 6151 | 7.36% | 5.66% | 5.74% | 0 | RedEarth Tx Mana
| 87 | 145830 | 63226 | 2306 | 5.28% | 4.30% | 4.42% | 0 | RedEarth I2C dri
This process is used for communication between stacked switches when programming qos entries and therefore if we have a link constantly flapping this would cause the "HRPC qos request" process to be invoked frequently leading to high cpu
--Shutting down gi4/0/5 resolved the issue
Dec 8 05:55:55.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up
Dec 8 05:55:56.933: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/5, changed state to up
Dec 8 05:55:59.189: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to down
Dec 8 05:55:59.550: %LINK-5-CHANGED: Interface FastEthernet0, changed state to administratively down
Dec 8 05:56:00.145: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/5, changed state to down
Dec 8 05:56:03.316: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up
Dec 8 05:57:53.098: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 94%/9%, Top 3 processes(Pid/Util): 383/24%, 214/20%, 190/20%
Dec 8 05:58:18.004: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to down
Dec 8 05:58:20.126: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up
As a work around could you please try to shut the interface gi4/0/5 and check the improvement in CPU.
- Gi4/0/5 was responsible for the high cpu utilization we were seeing due to the "HRPC qos request" process. This process is used for communication between stacked switches when programming qos entries and therefore if we have a link constantly flapping this would cause the "HRPC qos request" process to be invoked frequently leading to high cpu
--Shutting down gi4/0/5 resolved the issue
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide