Cisco 4500 - SUP-6E - HIGH CPU Utilization

Stepan Vesely · ‎08-11-2011

Hello cisco specialists,

I supervise on LAN network and it seems that cpu has high utilization - even at night, when nobody is at work!

Somebody with same problem or anybody who good give me some tips?

Sa Miss :

The behavior changed in Cisco IOS Releases 12.2(50)SG4, 12.2(53)SG, or later; packets are processed

in hardware and no extra copy is sent to the CPU. (CSCsu81046)

* I got there 12.2(54)SG1

Adj SameIf Fail :

Packets that is routed out of the same interface.

* Historically between this GW (4500) and ASA FW is VRF configuration...maybe?

Here is some show reports:

****

sh processes cpu sort

CPU utilization for five seconds: 80%/2%; one minute: 80%; five minutes: 80%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

57 319024866 1148670293 277 70.07% 70.23% 70.36% 0 Cat4k Mgmt LoPri

56 36493723 97272812 375 6.55% 6.06% 6.21% 0 Cat4k Mgmt HiPri

108 3678708 2290993 1605 2.07% 0.75% 0.40% 0 SpanTree Flush

121 5412953 2487404 2176 1.43% 0.99% 0.96% 0 Spanning Tree

****

sh platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

K5CpuMan Review 30.00 46.98 30 23 100 500 68 58 55 6002:30

K5ForerunnerPacketMa 1.50 26.16 4 0 100 500 35 29 24 2641:47

***

sh platform cpu packet statistics

RkiosSysPacketMan:

Packet allocation failures: 0

Packet Buffer(Software Common) allocation failures: 0

Packet Buffer(Software ESMP) allocation failures: 0

Packet Buffer(Software EOBC) allocation failures: 0

Packet Buffer(Software SupToSup) allocation failures: 0

IOS Packet Buffer Wrapper allocation failures: 0

Packets Dropped In Processing Overall

Total 5 sec avg 1 min avg 5 min avg 1 hour avg

-------------------- --------- --------- --------- ----------

34555740 4 15 25 15

Packets Dropped In Processing by CPU event

Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Sa Miss 34091682 3 14 24 15

L2 Router 524 0 0 0 0

Input Acl Fwd 467 0 0 0 0

Input ACl Copy 463067 0 0 0 0

Packets Dropped In Processing by Priority

Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Normal 464057 0 0 0 0

Medium 34091682 3 14 24 15

High 1 0 0 0 0

Packets Dropped In Processing by Reason

Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg

------------------ -------------------- --------- --------- --------- ----------

SrcAddrTableFilt 15 0 0 0 0

STPDrop 46 0 0 0 0

AclActionDrop 119 0 0 0 0

NoDstPorts 345 0 0 0 0

Tx Mode Drop 34554691 4 15 25 15

Extra packet due to RxProc hwAssisted Bridging 524 0 0 0 0

Total packet queues 64

Packets Received by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- - -------------- --------- --------- --------- ----------

Esmp 45142376 83 68 64 55

Input ACL fwd(snooping) 10066057 22 14 13 3

L2 bridge to CPU, 0 17114371 59 33 28 20

Host Learning 34073570 4 15 24 15

L2 Control 6913331 11 7 7 0

Input ACL log, unreach 463060 0 0 0 0

Ttl Expired 7282 0 0 0 0

Adj SameIf Fail 5699045684 2491 2328 2139 6378

L2 router to CPU, 7 507862 0 0 0 0

L3 Glean, 7 1044562 2 0 0 0

L3 Receive, 7 3581175 13 7 7 1

Packets Dropped by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

L2 bridge to CPU, 0 230 0 0 0 0

Host Learning 256610 0 0 0 0

Adj SameIf Fail 21900926 1 0 1 5

Packets Transmitted from CPU per Output Interface

Interface Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Gi2/1 49740 0 0 0 0

Gi2/2 42479 0 0 0 0

Gi2/5 94568 0 0 0 0

Gi2/6 93600 0 0 0 0

Gi2/9 52403 0 0 0 0

Gi2/11 93883 0 0 0 0

Gi2/13 94954 0 0 0 0

Gi2/14 477 0 0 0 0

Gi2/15 92711 0 0 0 0

Gi2/16 477 0 0 0 0

Gi2/17 93523 0 0 0 0

Gi2/18 93511 0 0 0 0

Gi4/1 92340 0 0 0 0

Gi4/2 1003 0 0 0 0

Gi4/14 8500 0 0 0 0

Gi4/15 8503 0 0 0 0

Gi4/16 93308 0 0 0 0

Gi4/17 8501 0 0 0 0

Gi4/19 1987 0 0 0 0

Gi4/21 8500 0 0 0 0

Gi4/22 8500 0 0 0 0

Gi4/23 92339 0 0 0 0

Gi4/24 3173 0 0 0 0

Gi4/25 3435 0 0 0 0

Gi4/26 3182 0 0 0 0

Gi4/27 9644 0 0 0 0

Gi4/28 3175 0 0 0 0

Gi4/29 3352 0 0 0 0

Gi4/30 3351 0 0 0 0

Gi4/31 8500 0 0 0 0

Gi4/33 92340 0 0 0 0

Gi4/35 3175 0 0 0 0

Gi4/36 3176 0 0 0 0

Gi4/37 93605 0 0 0 0

Gi4/38 3175 0 0 0 0

Gi4/39 3175 0 0 0 0

Gi4/40 3175 0 0 0 0

Gi4/42 3175 0 0 0 0

Gi4/46 92340 0 0 0 0

Gi6/1 52034 0 0 0 0

Gi6/2 4777 0 0 0 0

Gi6/3 8507 0 0 0 0

Gi6/4 3366 0 0 0 0

Gi6/5 8674 0 0 0 0

Gi6/6 8662 0 0 0 0

Gi6/7 4725 0 0 0 0

Gi6/8 41 0 0 0 0

Gi6/9 93308 0 0 0 0

Gi6/11 15952 0 0 0 0

Gi6/12 9546 0 0 0 0

Gi6/13 3443 0 0 0 0

Gi6/14 41 0 0 0 0

Gi6/15 41 0 0 0 0

Gi6/16 8506 0 0 0 0

Gi6/20 3863 0 0 0 0

Gi6/21 8761 0 0 0 0

Gi6/22 41 0 0 0 0

Gi6/24 8722 0 0 0 0

Gi6/28 3339 0 0 0 0

Gi6/29 8671 0 0 0 0

Gi6/30 8672 0 0 0 0

Gi6/31 8672 0 0 0 0

Gi6/32 3339 0 0 0 0

Gi6/33 92340 0 0 0 0

Gi6/34 56951 0 0 0 0

Gi6/35 1327 0 0 0 0

Gi6/36 2556 0 0 0 0

Gi6/37 92339 0 0 0 0

Gi6/39 8668 0 0 0 0

Gi6/40 3339 0 0 0 0

Gi6/45 4778 0 0 0 0

Gi6/46 93072 0 0 0 0

Gi6/47 1405 0 0 0 0

Gi6/48 1476 0 0 0 0

Te1/1 66054 0 0 0 0

Te1/2 52727 0 0 0 0

Packets Received at CPU per Input Interface

Interface Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Gi2/1 985 0 0 0 985

Gi2/2 701 0 0 0 701

Gi2/5 91755 0 0 0 91755

Gi2/6 9073 0 0 0 9073

Gi2/9 4147 0 0 0 4147

Gi2/11 23063 0 0 0 23063

Gi2/13 3488 0 0 0 3488

Gi2/14 3633 0 0 0 3633

Gi2/15 1048 0 0 0 1048

Gi2/16 123 0 0 0 123

Gi2/17 3188 0 0 0 3188

Gi2/18 12470 0 0 0 12470

Gi4/1 164 0 0 0 164

Gi4/2 2 0 0 0 2

Gi4/14 8 0 0 0 8

Gi4/15 2515 0 0 0 2515

Gi4/17 4 0 0 0 4

Gi4/21 4 0 0 0 4

Gi4/27 126 0 0 0 126

Gi4/28 6 0 0 0 6

Gi4/29 133 0 0 0 133

Gi4/30 132 0 0 0 132

Gi4/33 2440 0 0 0 2440

Gi4/35 36 0 0 0 36

Gi4/37 1113 0 0 0 1113

Gi4/38 2594 0 0 0 2594

Gi4/40 6 0 0 0 6

Gi4/42 2474 0 0 0 2474

Gi4/46 47 0 0 0 47

Gi6/1 28612 0 0 0 28612

Gi6/2 47 0 0 0 47

Gi6/3 149 0 0 0 149

Gi6/4 20 0 0 0 20

Gi6/5 24 0 0 0 24

Gi6/6 2451 0 0 0 2451

Gi6/7 27 0 0 0 27

Gi6/8 95 0 0 0 95

Gi6/11 28 0 0 0 28

Gi6/12 3078 0 0 0 3078

Gi6/13 202 0 0 0 202

Gi6/14 2 0 0 0 2

Gi6/15 8 0 0 0 8

Gi6/16 63 0 0 0 63

Gi6/20 707 0 0 0 707

Gi6/21 170 0 0 0 170

Gi6/22 16935 0 0 0 16935

Gi6/24 97 0 0 0 97

Gi6/29 350 0 0 0 350

Gi6/30 263 0 0 0 263

Gi6/31 506 0 0 0 506

Gi6/32 1 0 0 0 1

Gi6/33 2440 0 0 0 2440

Gi6/34 24333 0 0 0 24333

Gi6/35 534 0 0 0 534

Gi6/36 1691 0 0 0 1691

Gi6/39 9 0 0 0 9

Gi6/40 1 0 0 0 1

Gi6/45 32 0 0 0 32

Gi6/46 9564 0 0 0 9564

Gi6/47 1249 0 0 0 1249

Gi6/48 150 0 0 0 150

Te1/1 111091 0 0 0 111091

Te1/2 23546 0 0 0 23546

Thank you for any advice.

Best Regards,

Steve

Stepan Vesely · ‎08-11-2011

Hello all readers,

If somebody would like, now I know the answer on this question.

I do some tests with traffic at night...how?

On GW I do a monitoring - all incoming packets to CPU I mirror to PC (wireshark installed)

Code:

****************

conf t

monitor session 1 source cpu rx

monitor session 1 filter packet-type good rx

monitor session 1 destination interface gigabitEthernet X/Y

*****************

Find out, that some PC do a backup cross whole network to internet.

Have a nice day,

Steve

andtoth · ‎08-13-2011

Hi Stepan,

Thanks for sharing your findings, glad to hear you have found the cause of this.

For your reference, please have a look at the following documentation which will help you to identify the reason for high CPU usage on 4500 switches.

High CPU Utilization on Cisco IOS Software-Based Catalyst 4500 Switches

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

Best regards,

Andras

k.aumell · ‎10-16-2013

So is there actually a solution here? Was it just a PC sending a backup to the internet.

I have a pair of newly-installed 4500X's exhibiting the EXACT same symptoms as in the OP (SA MIss, Adj SameIf Fail), and don't see a root cause as yet.

Arumugam Muthaiah · ‎10-16-2013

Hi Aumell,

Adj SameIf Fail packets indicate that packets are being routed out of the same interface, so basically these packets are sent to the CPU for ICMP redirect. If an IP unicast packet is to be routed via the port in which the packet originally was received on, it may be sent to the CPU so that software sends ICMP message to the source host. Disabling the reception and transmission of certain types of messages on an interface can minimize the amount of CPU load that is required to process unrequired packets.

Suggestion:

Use the no ip redirects configuration command on all the SVIs or L3 interface

Refer:

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#redirects

Useful commands to troubleshoot High CPU,

#show processes cpu - in order to identify the Cisco IOS processes that consume CPU cycles

#show platform health - Check CPU usage on platform depending processes. If K5CpuMan Review, execute the below command identity the type of traffic that hits the CPU

#show platform cpu packet statistics - Check traffic on the CPU queue

It would be good to see what kind of traffic is currently hitting the CPU using internal inband capture tool:

#debug platform packet all buffer

#ahow platform cpu packet buffered

Note: This debug does not affect CPU utilization much.

.

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***