- Cisco Community
- Technology and Support
- Networking
- Switching
- High CPU usage for Catalyst 3750X on IP Input proccess
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
High CPU usage for Catalyst 3750X on IP Input proccess
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2015 06:04 AM - edited 03-08-2019 02:20 AM
Dear Community members,
I need your advise for troubleshooting a high CPU usage problem on a Catalyst 3750X L3 switch, which is the core one in my network. This problem is producing high latency and data lost that is driving my network slow and unstable.
So, the evidence. This is the output of "show proc cpu sorted" command
#show proc cpu sorted CPU utilization for five seconds: 98%/28%; one minute: 99%; five minutes: 99% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 214 41003600 13944778 2940 24.15% 25.21% 25.35% 0 IP Input 169 34067372 3928624 8671 16.31% 14.55% 14.37% 0 Hulc LED Process 232 20500311 2892288 7087 8.95% 8.43% 8.43% 0 Spanning Tree 125 19816534 4237944 4675 5.59% 7.24% 7.74% 0 hpm main process 12 3744640 2199415 1702 2.87% 2.56% 2.45% 0 ARP Input 85 1858596 630601 2947 1.27% 0.95% 0.89% 0 RedEarth Tx Mana 129 2207888 176060 12540 0.95% 1.04% 1.03% 0 hpm counter proc 212 2836209 795249 3566 0.79% 1.46% 1.32% 0 IP ARP Adjacency 53 137483 4075 33738 0.79% 0.09% 0.06% 0 Per-minute Jobs 245 608854 166106 3665 0.79% 0.64% 0.54% 0 PI MATM Aging Pr 121 68235 11682 5841 0.63% 0.07% 0.01% 0 Strider Tcam Mem 91 1099121 68040 16154 0.47% 0.52% 0.48% 0 Adjust Regions 340 471153 649411 725 0.31% 0.27% 0.23% 0 VLAN Manager 84 686262 849745 807 0.31% 0.34% 0.36% 0 RedEarth I2C dri 51 278373 3518118 79 0.31% 0.30% 0.28% 0 Net Input 170 398886 133716 2983 0.15% 0.14% 0.15% 0 HL3U bkgrd proce
As you can see, the CPU is completely utilized and the top process is IP Input. Googleing I learned that the idea is to use interrupt-level switching (Fast,CEF, between others) instead of proccess-level one, so there is no CPU payload on switching. Checking this...
#show cef interface brief Interface IP-Address Status Switching Vlan1 10.100.10.1 up no dCEF FastEthernet0 unassigned down no dCEF GigabitEthernet1/0/1 unassigned up CEF GigabitEthernet1/0/2 unassigned down CEF GigabitEthernet1/0/3 unassigned up CEF GigabitEthernet1/0/4 unassigned up CEF GigabitEthernet1/0/5 unassigned up CEF GigabitEthernet1/0/6 unassigned up CEF GigabitEthernet1/0/7 unassigned up CEF GigabitEthernet1/0/8 unassigned down CEF GigabitEthernet1/0/9 unassigned up CEF GigabitEthernet1/0/10 unassigned down CEF GigabitEthernet1/0/11 unassigned up CEF GigabitEthernet1/0/12 unassigned up CEF GigabitEthernet1/0/13 unassigned up CEF GigabitEthernet1/0/14 unassigned up CEF GigabitEthernet1/0/15 unassigned up CEF GigabitEthernet1/0/16 unassigned up CEF GigabitEthernet1/0/17 unassigned up CEF GigabitEthernet1/0/18 unassigned up CEF GigabitEthernet1/0/19 unassigned up CEF GigabitEthernet1/0/20 unassigned up CEF GigabitEthernet1/0/21 unassigned up CEF GigabitEthernet1/0/22 unassigned up CEF GigabitEthernet1/0/23 unassigned up CEF GigabitEthernet1/0/24 unassigned up CEF GigabitEthernet1/1/1 unassigned down CEF GigabitEthernet1/1/2 unassigned down CEF GigabitEthernet1/1/3 unassigned down CEF GigabitEthernet1/1/4 unassigned down CEF TenGigabitEthernet1/1/1 unassigned down CEF TenGigabitEthernet1/1/2 unassigned down CEF Null0 unassigned up no CEF Vlan2 X.X.171.126 up CEF Vlan3 unassigned up CEF Vlan4 unassigned up CEF Vlan6 X.X.165.65 up CEF Vlan7 X.X.166.190 up CEF Vlan8 X.X.152.2 up CEF Vlan9 X.X.166.62 up CEF Vlan10 unassigned up CEF Vlan12 X.X.161.126 up CEF Vlan13 X.X.171.254 up CEF Vlan14 X.X.167.254 up CEF Vlan15 X.X.166.65 up CEF Vlan16 unassigned up CEF Vlan17 X.X.167.62 up CEF Vlan19 X.X.161.190 up CEF Vlan21 X.X.167.126 up CEF Vlan22 X.X.171.158 up CEF Vlan27 X.X.162.226 up CEF Vlan28 X.X.162.242 up CEF Vlan30 10.201.6.67 up CEF Vlan60 192.168.60.1 up CEF Vlan61 unassigned up CEF Vlan100 192.168.100.1 up CEF Vlan200 unassigned up CEF Vlan201 unassigned up CEF Vlan240 X.X.168.253 up no dCEF Vlan250 X.X.171.190 up CEF Vlan251 192.168.101.1 up CEF Vlan252 192.168.102.1 up CEF Vlan253 192.168.103.30 up CEF Vlan260 X.X.162.190 up CEF Vlan280 unassigned up CEF Vlan281 unassigned down CEF Vlan298 unassigned down CEF Vlan301 unassigned down CEF StackPort1 unassigned down CEF Virtual1 unassigned up - Virtual2 unassigned up -
I can see that CEF is enabled on almost every interface, but checking "show interface switching" (output cropped for readability)
#show interface switching
Vlan1
Throttle count 0
Drops RP 1139 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 339540 75121303 57097 4611829
Cache misses 0 - - -
Fast 0 0 11 1053
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 1596666 95800784 72413 4344780
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan2
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2816137 201283125 10400 1470747
Cache misses 0 - - -
Fast 1205 183277 10 2542
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 132743 7964580 86383 5182980
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan3
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 68260 4095600 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2 120 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan4
All statistics for this interface are zero.
Vlan6
Throttle count 0
Drops RP 9 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 5140914 356471282 23015 2993665
Cache misses 0 - - -
Fast 428 39212 6 1135
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 13803 828180 16550 993000
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan7
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 151934 14734706 7539 1527065
Cache misses 0 - - -
Fast 156 14866 6 1315
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 10696 641760 9251 555060
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan8
Throttle count 0
Drops RP 53150 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 76358777 5367493319 759413 66953170
Cache misses 0 - - -
Fast 110151 10095563 74 13479
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 779028 46741694 459708 27582480
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 127 7620 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
... I notice that almost all the switching is done by process, so I don't know how to work around this.
To put into context. This switch connect directly or indirectly almost 190 other devices, between switches (2960s, 2950s, 3560, and 3750) and access points. It is the default gateway for a lot of VLAN and directly connect our WAN access.
The switch is running IOS 12.2(58)SE2. I have red that in other post (lik this one https://supportforums.cisco.com/discussion/11628666/cisco-3750x-24se-12258se2-cpu-utilization-high) that it's advice to to downgrade to 12.2(55)SE8 for stability reason, but this post is 3 years old so I'd like to know if this remains as a valid solution or is another preferable IOS version to work around this problem.
From the "show proc cpu" command I can see other 2 proccess eating CPU resource, HULC LED and Spanning Tree, but I'd like to troubleshoot the IP Input one first, cause is the top most one.
Please, advise for where should I look for solving this problem.
Thanks very much.
- Labels:
-
Other Switching
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-26-2015 05:59 AM
Hi yusuf habibi,
I already correct that, as Mark Malone pointed that out earlier in this post. Anyway, this didn't made IP Input process to decrease.
Thanks
- « Previous
- Next »
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide
