- Cisco Community
- Technology and Support
- Networking
- Switching
- If you are seeing a lot of
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
High CPU usage for Catalyst 3750X on IP Input proccess
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2015 06:04 AM - edited 03-08-2019 02:20 AM
Dear Community members,
I need your advise for troubleshooting a high CPU usage problem on a Catalyst 3750X L3 switch, which is the core one in my network. This problem is producing high latency and data lost that is driving my network slow and unstable.
So, the evidence. This is the output of "show proc cpu sorted" command
#show proc cpu sorted CPU utilization for five seconds: 98%/28%; one minute: 99%; five minutes: 99% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 214 41003600 13944778 2940 24.15% 25.21% 25.35% 0 IP Input 169 34067372 3928624 8671 16.31% 14.55% 14.37% 0 Hulc LED Process 232 20500311 2892288 7087 8.95% 8.43% 8.43% 0 Spanning Tree 125 19816534 4237944 4675 5.59% 7.24% 7.74% 0 hpm main process 12 3744640 2199415 1702 2.87% 2.56% 2.45% 0 ARP Input 85 1858596 630601 2947 1.27% 0.95% 0.89% 0 RedEarth Tx Mana 129 2207888 176060 12540 0.95% 1.04% 1.03% 0 hpm counter proc 212 2836209 795249 3566 0.79% 1.46% 1.32% 0 IP ARP Adjacency 53 137483 4075 33738 0.79% 0.09% 0.06% 0 Per-minute Jobs 245 608854 166106 3665 0.79% 0.64% 0.54% 0 PI MATM Aging Pr 121 68235 11682 5841 0.63% 0.07% 0.01% 0 Strider Tcam Mem 91 1099121 68040 16154 0.47% 0.52% 0.48% 0 Adjust Regions 340 471153 649411 725 0.31% 0.27% 0.23% 0 VLAN Manager 84 686262 849745 807 0.31% 0.34% 0.36% 0 RedEarth I2C dri 51 278373 3518118 79 0.31% 0.30% 0.28% 0 Net Input 170 398886 133716 2983 0.15% 0.14% 0.15% 0 HL3U bkgrd proce
As you can see, the CPU is completely utilized and the top process is IP Input. Googleing I learned that the idea is to use interrupt-level switching (Fast,CEF, between others) instead of proccess-level one, so there is no CPU payload on switching. Checking this...
#show cef interface brief Interface IP-Address Status Switching Vlan1 10.100.10.1 up no dCEF FastEthernet0 unassigned down no dCEF GigabitEthernet1/0/1 unassigned up CEF GigabitEthernet1/0/2 unassigned down CEF GigabitEthernet1/0/3 unassigned up CEF GigabitEthernet1/0/4 unassigned up CEF GigabitEthernet1/0/5 unassigned up CEF GigabitEthernet1/0/6 unassigned up CEF GigabitEthernet1/0/7 unassigned up CEF GigabitEthernet1/0/8 unassigned down CEF GigabitEthernet1/0/9 unassigned up CEF GigabitEthernet1/0/10 unassigned down CEF GigabitEthernet1/0/11 unassigned up CEF GigabitEthernet1/0/12 unassigned up CEF GigabitEthernet1/0/13 unassigned up CEF GigabitEthernet1/0/14 unassigned up CEF GigabitEthernet1/0/15 unassigned up CEF GigabitEthernet1/0/16 unassigned up CEF GigabitEthernet1/0/17 unassigned up CEF GigabitEthernet1/0/18 unassigned up CEF GigabitEthernet1/0/19 unassigned up CEF GigabitEthernet1/0/20 unassigned up CEF GigabitEthernet1/0/21 unassigned up CEF GigabitEthernet1/0/22 unassigned up CEF GigabitEthernet1/0/23 unassigned up CEF GigabitEthernet1/0/24 unassigned up CEF GigabitEthernet1/1/1 unassigned down CEF GigabitEthernet1/1/2 unassigned down CEF GigabitEthernet1/1/3 unassigned down CEF GigabitEthernet1/1/4 unassigned down CEF TenGigabitEthernet1/1/1 unassigned down CEF TenGigabitEthernet1/1/2 unassigned down CEF Null0 unassigned up no CEF Vlan2 X.X.171.126 up CEF Vlan3 unassigned up CEF Vlan4 unassigned up CEF Vlan6 X.X.165.65 up CEF Vlan7 X.X.166.190 up CEF Vlan8 X.X.152.2 up CEF Vlan9 X.X.166.62 up CEF Vlan10 unassigned up CEF Vlan12 X.X.161.126 up CEF Vlan13 X.X.171.254 up CEF Vlan14 X.X.167.254 up CEF Vlan15 X.X.166.65 up CEF Vlan16 unassigned up CEF Vlan17 X.X.167.62 up CEF Vlan19 X.X.161.190 up CEF Vlan21 X.X.167.126 up CEF Vlan22 X.X.171.158 up CEF Vlan27 X.X.162.226 up CEF Vlan28 X.X.162.242 up CEF Vlan30 10.201.6.67 up CEF Vlan60 192.168.60.1 up CEF Vlan61 unassigned up CEF Vlan100 192.168.100.1 up CEF Vlan200 unassigned up CEF Vlan201 unassigned up CEF Vlan240 X.X.168.253 up no dCEF Vlan250 X.X.171.190 up CEF Vlan251 192.168.101.1 up CEF Vlan252 192.168.102.1 up CEF Vlan253 192.168.103.30 up CEF Vlan260 X.X.162.190 up CEF Vlan280 unassigned up CEF Vlan281 unassigned down CEF Vlan298 unassigned down CEF Vlan301 unassigned down CEF StackPort1 unassigned down CEF Virtual1 unassigned up - Virtual2 unassigned up -
I can see that CEF is enabled on almost every interface, but checking "show interface switching" (output cropped for readability)
#show interface switching
Vlan1
Throttle count 0
Drops RP 1139 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 339540 75121303 57097 4611829
Cache misses 0 - - -
Fast 0 0 11 1053
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 1596666 95800784 72413 4344780
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan2
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2816137 201283125 10400 1470747
Cache misses 0 - - -
Fast 1205 183277 10 2542
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 132743 7964580 86383 5182980
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan3
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 68260 4095600 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2 120 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan4
All statistics for this interface are zero.
Vlan6
Throttle count 0
Drops RP 9 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 5140914 356471282 23015 2993665
Cache misses 0 - - -
Fast 428 39212 6 1135
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 13803 828180 16550 993000
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan7
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 151934 14734706 7539 1527065
Cache misses 0 - - -
Fast 156 14866 6 1315
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 10696 641760 9251 555060
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan8
Throttle count 0
Drops RP 53150 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 76358777 5367493319 759413 66953170
Cache misses 0 - - -
Fast 110151 10095563 74 13479
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 779028 46741694 459708 27582480
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 127 7620 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
... I notice that almost all the switching is done by process, so I don't know how to work around this.
To put into context. This switch connect directly or indirectly almost 190 other devices, between switches (2960s, 2950s, 3560, and 3750) and access points. It is the default gateway for a lot of VLAN and directly connect our WAN access.
The switch is running IOS 12.2(58)SE2. I have red that in other post (lik this one https://supportforums.cisco.com/discussion/11628666/cisco-3750x-24se-12258se2-cpu-utilization-high) that it's advice to to downgrade to 12.2(55)SE8 for stability reason, but this post is 3 years old so I'd like to know if this remains as a valid solution or is another preferable IOS version to work around this problem.
From the "show proc cpu" command I can see other 2 proccess eating CPU resource, HULC LED and Spanning Tree, but I'd like to troubleshoot the IP Input one first, cause is the top most one.
Please, advise for where should I look for solving this problem.
Thanks very much.
- Labels:
-
Other Switching
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2015 06:49 AM
Hi Dago,
You could also configure Dynamic ARP inspection on the switch to overcome this situation. Please refer to the document in the link below:
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1003452
Following is the link to the config guie of this feature:
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750x_3560x/software/release/12-2_55_se/configuration/guide/3750xscg/swdynarp.html
HTH,
Roopa
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2015 06:55 AM
Hi Martin,
I dont think broadcast storm control rate limits CPU bound traffic like ARP. I think it does it for data traffic.
Another feature apart from DAI Referenced above , that can be used to ratelimit arp is the protocol storm protection feature. Please take a look at the link below.
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750x_3560x/software/release/12-2_58_se/configuration/guide/3750xscg/swtrafc.html#pgfId-1173343
Regards,
Roopa
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2015 08:30 AM
How many link flapping messages are you getting? A few from wireless clients (or even wired ones) isn't an issue, but a lot might indicate a loop that STP can't resolve. I've seen a hub accidentally connected to 2 separate network ports that can cause serious problems.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2015 10:02 AM
Guys, thanks for the advice for ARP issue resolution, but I must say that somehow I made a mistake (or wireshark had some trouble polting data .... don't know), but after notice some strange behavior whit plotted IO graph (some values remained the same even after changing it filter), I reopened wireshark and reloaded capture file and then ARP package wasn't as high as I reported. Atached is the Protocl Hierarchy which show very few ARP traffic and a lot of RTSP (from IP surviliance cameras .... will check on that latter).
How many link flapping messages are you getting? A few from wireless clients (or even wired ones) isn't an issue, but a lot might indicate a loop that STP can't resolve. I've seen a hub accidentally connected to 2 separate network ports that can cause serious problems
Sometimes there is no MACFLAP messages, they aren't constant, but if I do a "show log" mostly the only thing I can see are those one. They all came from wifi clients. There is no HUB on our network, just switches, all with SPT capability.
Will keep on searching for the couse of my problem.
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2015 06:18 PM
It's worth checking where all the RSTP packets are coming from.
If the TCNs are coming from ports connected to end clients then make sure you have enabled "spanning-tree portfast" for access ports.
But don't enable it on any switch to switch connections.
If there a lot of TCNs being generated, for whatever reason, then this will cause your switch to flush it's mac addresses tables which will mean traffic is flooded until the mac addresses are learnt again.
Jon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-09-2015 12:29 AM
You can use this commanmd to help narrow down where STP loops are coming from in a Cisco network in case thats the issue
sh spanning-tree detail | i ieee|occur|from|is exec
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-13-2015 11:41 AM
So .... I still running this problem and bosses are starting to press me up :(
What we have done and notice this las tieme
We did a IOS downgrade from 12.2(58)SE2 to 12.2.(55)SE10. Althought we noticed a "base" consumption decrease during non-work hours, didn't solve the issue, couse during work hours CPU hit 99-100% utilization, so we assume this is not a IOS bug problem.
After downgrading we started geting this log message.
Nov 13 18:22:41.462: %PLATFORM_ENV-1-FRU_PS_ACCESS: FRU Power Supply is not responding
I know this is not related to CPU utilization, but if anyone can point out what is about. Some guy on another forum advise to upgrade to IOS v15 becouse of this message ¬¬U
Checking Input Queue Drops I'm getting this
#show controllers cpu-interface
.
.
.
cpu-queue-frames retrieved dropped invalid hol-block stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc 0 0 0 0 0
stp 8734828 0 0 0 0
ipc 0 0 0 0 0
routing protocol 361277676 0 0 0 0
L2 protocol 715921 0 0 0 0
remote console 0 0 0 0 0
sw forwarding 1105974 0 0 0 0
host 65489122 0 0 0 0
broadcast 145536630 0 0 0 0
cbt-to-spt 0 0 0 0 0
igmp snooping 219056228 0 0 0 0
icmp 186191930 0 0 0 0
logging 0 0 0 0 0
rpf-fail 0 0 0 0 0
dstats 0 0 0 0 0
cpu heartbeat 5453326 0 0 0 0
cpu-queue static inuse static added
----------------- ------------ ------------
rpc 0 0
stp 0 0
ipc 0 0
routing protocol 0 0
L2 protocol 0 0
remote console 0 0
sw forwarding 0 0
host 0 0
broadcast 0 0
cbt-to-spt 0 0
igmp snooping 0 0
icmp 0 0
logging 0 0
rpf-fail 0 0
dstats 0 0
cpu heartbeat 0 0
#show buffers
Buffer elements:
1061 in free list (500 max allowed)
237838781 hits, 0 misses, 1024 created
Public buffer pools:
Small buffers, 104 bytes (total 84, permanent 50, peak 209 @ 3d05h):
78 in free list (20 min, 150 max allowed)
115746099 hits, 95 misses, 227 trims, 261 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 34, permanent 25, peak 177 @ 3d03h):
25 in free list (10 min, 150 max allowed)
1930057 hits, 752 misses, 369 trims, 378 created
125 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50, peak 77 @ 3d05h):
50 in free list (5 min, 150 max allowed)
1318164 hits, 9 misses, 27 trims, 27 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 16, permanent 10, peak 16 @ 3d05h):
1 in free list (0 min, 100 max allowed)
73 hits, 3 misses, 28 trims, 34 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Interface buffer pools:
Syslog ED Pool buffers, 600 bytes (total 132, permanent 132):
100 in free list (132 min, 132 max allowed)
227879 hits, 33 misses
RxQ1 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
8002569 hits, 774927 fallbacks
RxQ2 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
128 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
RxQ3 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
242385008 hits, 119511351 fallbacks
RxQ4 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
716488 hits, 21290530 misses
RxQ5 buffers, 2040 bytes (total 128, permanent 128):
64 in free list (0 min, 128 max allowed)
64 hits, 0 misses
RxQ6 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
1106795 hits, 1179012 misses
RxQ7 buffers, 2040 bytes (total 192, permanent 192):
55 in free list (0 min, 192 max allowed)
65557510 hits, 5927947 misses
RxQ8 buffers, 2040 bytes (total 64, permanent 64):
0 in free list (0 min, 64 max allowed)
145678354 hits, 274164627 misses
RxQ9 buffers, 2040 bytes (total 1, permanent 1):
0 in free list (0 min, 1 max allowed)
1 hits, 0 misses
RxQ10 buffers, 2040 bytes (total 64, permanent 64):
1 in free list (0 min, 64 max allowed)
215847135 hits, 3575586 fallbacks
RxQ11 buffers, 2040 bytes (total 16, permanent 16):
0 in free list (0 min, 16 max allowed)
186217703 hits, 186383864 misses
RxQ12 buffers, 2040 bytes (total 96, permanent 96):
0 in free list (0 min, 96 max allowed)
96 hits, 0 misses
RxQ13 buffers, 2040 bytes (total 16, permanent 16):
0 in free list (0 min, 16 max allowed)
16 hits, 0 misses
RxQ15 buffers, 2040 bytes (total 4, permanent 4):
0 in free list (0 min, 4 max allowed)
5456770 hits, 5456766 misses
HRPCRespFallback buffers, 1500 bytes (total 80, permanent 80):
80 in free list (0 min, 80 max allowed)
0 hits, 0 misses
RxQFB buffers, 2040 bytes (total 904, permanent 904):
858 in free list (0 min, 904 max allowed)
123328761 hits, 534252 misses
RxQ0 buffers, 2040 bytes (total 1200, permanent 1200):
700 in free list (0 min, 1200 max allowed)
500 hits, 0 misses
IPC buffers, 2048 bytes (total 300, permanent 300):
296 in free list (150 min, 500 max allowed)
36908 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
Jumbo buffers, 9240 bytes (total 200, permanent 200):
200 in free list (0 min, 200 max allowed)
0 hits, 0 misses
Header pools:
I can see that there is a lot of droped (misses) packages on almost any queue.
#show int | include Vlan|Input|Giga
Vlan1 is up, line protocol is up
Input queue: 1/75/5/0 (size/max/drops/flushes); Total output drops: 0
Vlan2 is up, line protocol is up
Input queue: 10/75/173/0 (size/max/drops/flushes); Total output drops: 0
Vlan3 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan4 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan6 is up, line protocol is up
Input queue: 17/75/4/0 (size/max/drops/flushes); Total output drops: 0
Vlan7 is up, line protocol is up
Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan8 is up, line protocol is up
Input queue: 166/75/37691/0 (size/max/drops/flushes); Total output drops: 0
Vlan9 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan10 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan12 is up, line protocol is up
Input queue: 11/75/3/0 (size/max/drops/flushes); Total output drops: 0
Vlan13 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan14 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan15 is up, line protocol is up
Input queue: 8/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan16 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan17 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan19 is up, line protocol is up
Input queue: 7/75/864/0 (size/max/drops/flushes); Total output drops: 0
Vlan21 is up, line protocol is up
Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan22 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan27 is up, line protocol is up
Input queue: 34/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan28 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan30 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan60 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan61 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan100 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan200 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan201 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan240 is administratively down, line protocol is down
Input queue: 0/75/17/0 (size/max/drops/flushes); Total output drops: 0
Vlan250 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan251 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan252 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan253 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan260 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan280 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan281 is down, line protocol is down
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan298 is down, line protocol is down
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan301 is down, line protocol is down
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8481 (bia 10f3.11ca.8481)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/2 is administratively down, line protocol is down (disabled)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8482 (bia 10f3.11ca.8482)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8483 (bia 10f3.11ca.8483)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/4 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8484 (bia 10f3.11ca.8484)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/5 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8485 (bia 10f3.11ca.8485)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/6 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8486 (bia 10f3.11ca.8486)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/7 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8487 (bia 10f3.11ca.8487)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/8 is administratively down, line protocol is down (disabled)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8488 (bia 10f3.11ca.8488)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/9 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8489 (bia 10f3.11ca.8489)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 31817
GigabitEthernet1/0/10 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is 10f3.11ca.848a (bia 10f3.11ca.848a)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 55
GigabitEthernet1/0/11 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.848b (bia 10f3.11ca.848b)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/12 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.848c (bia 10f3.11ca.848c)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/13 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.848d (bia 10f3.11ca.848d)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/14 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.848e (bia 10f3.11ca.848e)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 24708
GigabitEthernet1/0/15 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.848f (bia 10f3.11ca.848f)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/16 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8490 (bia 10f3.11ca.8490)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/17 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8491 (bia 10f3.11ca.8491)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/18 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8492 (bia 10f3.11ca.8492)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/19 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8493 (bia 10f3.11ca.8493)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/20 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8494 (bia 10f3.11ca.8494)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/21 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8495 (bia 10f3.11ca.8495)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3005
GigabitEthernet1/0/22 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8496 (bia 10f3.11ca.8496)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/0/23 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8497 (bia 10f3.11ca.8497)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 56
GigabitEthernet1/0/24 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10f3.11ca.8498 (bia 10f3.11ca.8498)
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
I can see that VLAN 8 and G1/0/14 are getting a lot of dorps
You can use this commanmd to help narrow down where STP loops are coming from in a Cisco network in case thats the issue
sh spanning-tree detail | i ieee|occur|from|is exec
#sh spanning-tree detail | i ieee|occur|from|is exec
VLAN0001 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 116 last change occurred 00:19:30 ago
from GigabitEthernet1/0/1
VLAN0002 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:30 ago
from GigabitEthernet1/0/1
VLAN0003 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 52 last change occurred 00:19:30 ago
from GigabitEthernet1/0/1
VLAN0004 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:30 ago
from GigabitEthernet1/0/1
VLAN0005 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:31 ago
from GigabitEthernet1/0/1
VLAN0006 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:31 ago
from GigabitEthernet1/0/1
VLAN0007 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:31 ago
from GigabitEthernet1/0/1
VLAN0008 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:31 ago
from GigabitEthernet1/0/1
VLAN0009 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:31 ago
from GigabitEthernet1/0/1
VLAN0010 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:31 ago
from GigabitEthernet1/0/1
VLAN0012 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 115 last change occurred 00:19:32 ago
from GigabitEthernet1/0/1
VLAN0013 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 55 last change occurred 00:19:32 ago
from GigabitEthernet1/0/1
VLAN0014 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:32 ago
from GigabitEthernet1/0/1
VLAN0015 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:32 ago
from GigabitEthernet1/0/1
VLAN0016 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 77 last change occurred 00:19:32 ago
from GigabitEthernet1/0/1
VLAN0017 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:32 ago
from GigabitEthernet1/0/1
VLAN0018 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:32 ago
from GigabitEthernet1/0/1
VLAN0019 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 39 last change occurred 00:19:33 ago
from GigabitEthernet1/0/1
VLAN0020 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 31 last change occurred 04:02:05 ago
from GigabitEthernet1/0/7
VLAN0021 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:33 ago
from GigabitEthernet1/0/1
VLAN0022 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:33 ago
from GigabitEthernet1/0/1
VLAN0023 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 54 last change occurred 00:19:34 ago
from GigabitEthernet1/0/1
VLAN0024 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 12 last change occurred 00:19:34 ago
from GigabitEthernet1/0/1
VLAN0027 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:34 ago
from GigabitEthernet1/0/1
VLAN0028 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 34 last change occurred 00:19:34 ago
from GigabitEthernet1/0/1
VLAN0030 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:35 ago
from GigabitEthernet1/0/1
VLAN0040 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:35 ago
from GigabitEthernet1/0/1
VLAN0051 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 35 last change occurred 00:19:35 ago
from GigabitEthernet1/0/1
VLAN0060 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:35 ago
from GigabitEthernet1/0/1
VLAN0061 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 35 last change occurred 00:19:35 ago
from GigabitEthernet1/0/1
VLAN0062 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 35 last change occurred 00:19:35 ago
from GigabitEthernet1/0/1
VLAN0063 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 34 last change occurred 00:19:35 ago
from GigabitEthernet1/0/1
VLAN0100 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:35 ago
from GigabitEthernet1/0/1
VLAN0200 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 97 last change occurred 00:19:36 ago
from GigabitEthernet1/0/1
VLAN0201 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 84 last change occurred 00:19:36 ago
from GigabitEthernet1/0/1
VLAN0202 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 83 last change occurred 00:19:36 ago
from GigabitEthernet1/0/1
VLAN0203 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 85 last change occurred 00:19:36 ago
from GigabitEthernet1/0/1
VLAN0204 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:37 ago
from GigabitEthernet1/0/1
VLAN0205 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:37 ago
from GigabitEthernet1/0/1
VLAN0206 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:37 ago
from GigabitEthernet1/0/1
VLAN0210 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 203 last change occurred 00:19:37 ago
from GigabitEthernet1/0/1
VLAN0211 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 111 last change occurred 00:19:38 ago
from GigabitEthernet1/0/1
VLAN0213 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 39 last change occurred 00:19:38 ago
from GigabitEthernet1/0/1
VLAN0214 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:38 ago
from GigabitEthernet1/0/1
VLAN0215 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:38 ago
from GigabitEthernet1/0/1
VLAN0220 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:38 ago
from GigabitEthernet1/0/1
VLAN0230 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:38 ago
from GigabitEthernet1/0/1
VLAN0237 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:38 ago
from GigabitEthernet1/0/1
VLAN0238 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:39 ago
from GigabitEthernet1/0/1
VLAN0240 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 45 last change occurred 00:19:39 ago
from GigabitEthernet1/0/1
VLAN0241 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 32 last change occurred 00:19:39 ago
from GigabitEthernet1/0/1
VLAN0242 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:39 ago
from GigabitEthernet1/0/1
VLAN0250 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:39 ago
from GigabitEthernet1/0/1
VLAN0251 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:40 ago
from GigabitEthernet1/0/1
VLAN0252 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 53 last change occurred 00:19:40 ago
from GigabitEthernet1/0/1
VLAN0253 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:40 ago
from GigabitEthernet1/0/1
VLAN0260 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:40 ago
from GigabitEthernet1/0/1
VLAN0280 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 35 last change occurred 00:19:40 ago
from GigabitEthernet1/0/1
VLAN0290 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 36 last change occurred 00:19:40 ago
from GigabitEthernet1/0/1
VLAN0291 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:40 ago
from GigabitEthernet1/0/1
VLAN0292 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 37 last change occurred 00:19:41 ago
from GigabitEthernet1/0/1
VLAN0299 is executing the ieee compatible Spanning Tree protocol
Number of topology changes 38 last change occurred 00:19:41 ago
from GigabitEthernet1/0/1
What I can see with this output is that there is a lot of TCN and most of them come from G1/0/7 which happen to be the interface that generate most of the CPU utilization (if I shutdown the interface CPU goes down).
If the TCNs are coming from ports connected to end clients then make sure you have enabled "spanning-tree portfast" for access ports.
But don't enable it on any switch to switch connections
That's what we do :)
Any advice on how can I proceede?
Thanks ;)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-13-2015 02:00 PM
You need to run that spanning-tree command a few times to see if the TCN count is increasing and how fast.
All the TCNs are coming from the same port.
If it is a switch connected to that port you need to run the same command on there and work out what is generating them.
It may not be STP but if you are seeing a lot of RSTP traffic you need to work out why.
Jon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-18-2015 11:13 PM
Dago
I think that becouse of so high level of CPU and Interrupts,your 3750 has no ability to send BPDUs to other devices in correct timing periods,or worse maybe missed to send some.If there is 100% link utilization on some ports,then some BPDUs cant reach lower levels.And ther you go,then all of that TCN are just consequence to that.
But first of all,you must decide which device was suppose to be--root--in your network and on that device you must set highest priority.
You must find out,which traffic is compromise your CPU,after lowering those CPU,it will maybe everything settle down include those TCNs.
I would suggest to use these commands
show controllers utilization to see from where is coming most of traffic.If you see 80-100% utilization on some interface,then use wireshark and analyize traffic.Use Statistic--IOgraph and in filter line wrote arp,icmp,stp
show mac address-table count | inc Available to see how many MACs are in table
clear counters
and then use
show interfaces accounting | inc ARP|Vlan to see how many ARPs are on some interfaces.And IF there is a lot of them on some explicite interface,then look with wireshark in these interface and see who is the source of them,and then find out why.
If there is problem with ICMP which may be very annoying,you can discover that on Controllers Queues.I wrote down this in my upper post,where I mention Debug.Whith debug tool,you must be very carefull and follow those instruction line by line.They are writen in those cisco article in my post,below your post.
I hope that you will soon find out what is happening in your network and solve the problem
BR
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-13-2015 07:47 PM
Hi Dago,
i think, you need verify / reconfiguring STP and on all switches. make sure your network having root of the STP. After you decide on the root switch, set the appropriate variables to designate the switch as the root switch. The only variable that you must set is the bridge priority. If the switch has a bridge priority that is lower than all the other switches, the other switches automatically select the switch as the root switch.
Note: Remember that one root switch is identified per-VLAN
You can also issue the set spantree portfast command, on a per-port basis.However, do not use this command when you have switch-to-switch connection. In this case, the command can result in a loop.
based on log spanning-tree details, you have a lot of number the topology changes.STP Path Cost Automatically Changes When a Port Speed/Duplex Is Changed. STP calculates the path cost based on the media speed (bandwidth) of the links between switches and the port cost of each port forwarding frame. Spanning tree selects the root port based on the path cost. The port with the lowest path cost to the root bridge becomes the root port. The root port is always in the forwarding state.
If the speed/duplex of the port is changed, spanning tree recalculates the path cost automatically. A change in the path cost can change the spanning tree topology.
Recommendation, set manually speed / duplex on each all interface.
for analysis, go to output interpreter tool cisco. put your show tech log before.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-18-2015 03:25 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2015 06:47 AM
Hello
You might want to configure broadcast storm-control on the port where you're receiving the ARP queries, to limit the number of broadcast ARP queries that you have to process. Have a look at this link: http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750x_3560x/software/release/12-2_53_se/configuration/guide/3750xscg/swtrafc.html#wp1063295
Best regards,
Martin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2015 06:54 AM
Ok Leo. Thanks. I'll consider this downgrade. What do you think about 15.0(2)SE8? As roor suggested that one.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2015 02:38 PM
What do you think about 15.0(2)SE8?
I can't comment on 15.0(2)SE8 running on 3750X. 15.0(2)SE8 works well with 2960S and I'm testing this version on 3560CG but I've been burnt with 15.0(2)SE train before plus the 3750X sits in a site distribution level. I ain't going to risk it again.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-23-2015 08:50 PM
Hi Dago,
should be ip route-cache enabled on interface vlan 240. for this case if you disable ip route-cache maybe impact to CPU. i had experience with same case before, and then i trying to enable ip route-cache on each interface (by default the services is enabled on interface).
why you should enabled "ip route-cache" on interface because on your switch for "ip routing" activated.
my assumption, if you want layer 3 services on your switches. Do not disable ip route-cache on each interface.
try this on interface vlan 240
interface Vlan240 ip address X.X.168.253 255.255.255.0 default ip route-cache cef default ip route-cache
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide
Make a few more discovery with command which you already use--show controllers cpu-interface
Then compare the counters result on each Queue--stp,icmp,hosts etc
Where you see a lot of counter differences,that is the reason for a higher CPU&Interrupts--but you could have a few of them.Go step by step.
When you see what is cause your problem go for Debug.
It,s all well writen in this article and it always help me to solve the high cpu problems on my 3750G or X.
Article.Read carefuly and go step by step--you cant fail.
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html