High CPU utilization on 3750x core

[Bruce Reed]

We're seeing high CPU utilization on a 3750x core switch. A few days ago this issue was more critical as we were seeing CPU averaging 80+%, with many spikes in the 90's. After some cleanup we're now averaging between 40-50%, with spikes rarely over that except on a config change. I've seen different posts on what's normal (and I understand this varies with a lot of factors,) but with a rather small network and only 15 vlans in operation I would think we should be averaging under 30%. 

Aside from IP Input the biggest culprit was the HULC LED Process. This was consuming between 20-30% of CPU. Disabling unused ports brought that down dramatically. I've read there were a few bugs with HULC LED in 12.2 and I'm wondering if that's true for 12.2(58)SE2, what were running. Would switching to a newer train, perhaps 15.02 help?

I've reviewed the troubleshooting notes on high CPU in 3750 and not much else really applies. We are using the desktop default template and I'm wondering whether we should switch to the routing template. There are 8 vlan svi's defined on the core, but that seems to be within the specs of the default template. 

What else should I look at? Or is 40-50% normal for this switch in this context?

Config, CPU proc output and tcam data below.

------------

core01-2f#show proc cpu sort

CPU utilization for five seconds: 45%/4%; one minute: 44%; five minutes: 45%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 214    85287582    18632071       4577 17.12% 17.96% 18.01%   0 IP Input
  85     9065762     1699130       5335  3.52%  3.63%  3.52%   0 RedEarth Tx Mana
  84     6967547     2551694       2730  2.71%  2.49%  2.47%   0 RedEarth I2C dri
 169    29617616     5593331       5295  2.40%  2.20%  2.25%   0 Hulc LED Process
  10      508550        3888     130799  2.24%  0.27%  0.18%   0 Licensing Auto U
 129     4921826      490654      10031  2.08%  1.86%  1.85%   0 hpm counter proc
 200      393946      144833       2720  1.91%  0.21%  0.13%   0 CDP Protocol
 330         491         343       1431  1.59%  0.12%  0.02%   3 SSH Process
 232     5653909     2957708       1911  0.79%  1.35%  1.38%   0 Spanning Tree
 181      995258       46468      21418  0.47%  0.37%  0.36%   0 HQM Stack Proces
 372      115197        3952      29149  0.47%  0.06%  0.00%   0 OBFL VOLT obfl0
  12      620030      791803        783  0.15%  0.29%  0.34%   0 ARP Input
 371       18821      100566        187  0.15%  0.01%  0.00%   0 LACP Protocol
 304      109925     2170801         50  0.15%  0.03%  0.00%   0 MDFS RP process
 125      915649     3677529        248  0.15%  0.26%  0.21%   0 hpm main process
 380       26065      236119        110  0.15%  0.01%  0.00%   0 NTP
  43      114380      276712        413  0.15%  0.02%  0.00%   0 Net Background
 182      513071      185491       2766  0.15%  0.15%  0.16%   0 HRPC qos request
 170      163776      170015        963  0.15%  0.07%  0.03%   0 HL3U bkgrd proce
  54      315883      234139       1349  0.15%  0.04%  0.05%   0 Per-Second Jobs
  20         133        3890         34  0.00%  0.00%  0.00%   0 IPC Dynamic Cach
  21           0           1          0  0.00%  0.00%  0.00%   0 IPC Session Serv
  19        1680       46493         36  0.00%  0.00%  0.00%   0 IPC Event Notifi
  24        7519      225319         33  0.00%  0.00%  0.00%   0 IPC Deferred Por
  22           0           1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager
  18           0           1          0  0.00%  0.00%  0.00%   0 IFS Agent Manage
  27         357       13349         26  0.00%  0.00%  0.00%   0 IPC Check Queue
  23        8531      225324         37  0.00%  0.00%  0.00%   0 IPC Periodic Tim
  17          58          14       4142  0.00%  0.00%  0.00%   0 Entity MIB API
  30         817       23381         34  0.00%  0.00%  0.00%   0 IPC Keep Alive M
  31        3862       46651         82  0.00%  0.00%  0.00%   0 IPC Loadometer
  32          42           4      10500  0.00%  0.00%  0.00%   0 PrstVbl

...

core01-2f#show proc cpu his

      444444433333222226666633333333334444444444444443333333333444

      889999999999444440000066666888884444444444444447777733333555
  100
   90
   80
   70
   60                  *****
   50 *******          *****                                   *
   40 ************     ***********************************     *
   30 ************     *****************************************
   20 **********************************************************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

      778997667577876666657786666665868676667676858676776685576777
      311980860562434804128137735625281303971576589537847638807039
  100    **
   90    **                                     * *
   80   ***     * *       * *       * *       * * *   *   *
   70 ********* **** *    ***** **  *** * ******* *********  ***
   60 ***#*************** **************************************
   50 ***##***************************************#*************
   40 ##########################################################
   30 ##########################################################
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                     1       1           1       1       1       1
      9999999999999980999999909999999999909999999099999990999999909999
      0999548985694580511939709899999999909999999099899990999999909999
  100  **** ****** * **  * *******************************************
   90 ***************************#************************************
   80 **************************######################################
   70 **********************##########################################
   60 ***#########*###*****###########################################
   50 **##############################################################
   40 ################################################################
   30 ################################################################
   20 ################################################################
   10 ################################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

core01-2f#show platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used
                                                                     Masks/Values    Masks/values

 Unicast mac addresses:                       6364/6364        953/953
 IPv4 IGMP groups + multicast routes:         1120/1120          1/1
 IPv4 unicast directly-connected routes:      6144/6144        563/563
 IPv4 unicast indirectly-connected routes:    2048/2048         67/67
 IPv4 policy based routing aces:               452/452          12/12
 IPv4 qos aces:                                512/512          21/21
 IPv4 security aces:                           964/964          36/36

!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname core01-2f
!
boot-start-marker
boot-end-marker
!
no logging console
enable secret 5 $1$yfafs$MddjQByZ3TC34FVxCLR5/
!
username XXXXXXX secret 5 $1$Q0awf$2347hv8Q/rBxtyXPjyG.
aaa new-model
!
aaa group server radius NPS
 server 10.16.72.6 auth-port 1812 acct-port 1813
!
aaa authentication login userAuthentication local group NPS
aaa authorization exec userAuthorization local group NPS if-authenticated
aaa authorization network userAuthorization local group NPS
aaa accounting exec default start-stop group NPS
aaa accounting system default start-stop group NPS
!
aaa session-id common
clock timezone PST -8 0
clock summer-time PDT recurring
switch 1 provision ws-c3750x-48p
switch 2 provision ws-c3750x-48p
system mtu routing 1500
no ip source-route
ip routing
!
no ip domain-lookup
vtp domain CUP3
vtp mode transparent
!
crypto pki trustpoint TP-self-signed-1203818496
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1203818496
 revocation-check none
 rsakeypair TP-self-signed-1203818496

!
crypto pki certificate chain TP-self-signed-1203818496
 certificate self-signed 01
  30820245 ......
        quit
!
spanning-tree mode pvst
spanning-tree extend system-id
spanning-tree vlan 1,11-12,21,50,901,903-907,909-911,915-916,920 priority 24576
spanning-tree vlan 921-923,925,930 priority 24576
!
port-channel load-balance src-dst-ip
!
vlan internal allocation policy ascending
!
vlan 11
 name data
!
vlan 12
 name domain-test
!
vlan 21
 name voice
vlan 50
 name DMZ
vlan 901
  name native

vlan 903
 name GuestWLAN
vlan 904
 name wireless
vlan 905
 name Trust
vlan 906
 name SSL-Int
vlan 907
 name SSL-Ext
vlan 909
 name Untrust
vlan 910
 name QA-Ext
vlan 911
 name Bonjour
vlan 915
 name Outside-VLAN-A
vlan 916
 name Outside-VLAN-B
vlan 920
 name NAT-Lab-A
vlan 921
 name NAT-Lab-B
vlan 922
  name NAT-Lab-C
vlan 923

 name NAT-Lab-D
vlan 925
 name Perf-Test
vlan 930
 name domain-test-wlan
!
interface Port-channel5
 description U/L to wlc-01
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 spanning-tree link-type point-to-point
!
interface Port-channel19
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 spanning-tree link-type point-to-point
!
interface Port-channel20
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
!
interface Port-channel21
 description U/L to sw02-2f
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 spanning-tree link-type point-to-point
!
interface Port-channel22

 description U/L to sw01-1f PO22
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 spanning-tree link-type point-to-point
!
interface FastEthernet0
 no ip address
 no ip route-cache cef
 no ip route-cache
!
interface GigabitEthernet1/0/1
 description TrustLAN
 switchport access vlan 905
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/2
 description TrustLAN
 switchport access vlan 905
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/3
 description TrustLAN
 switchport access vlan 905
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/4
 description TrustLAN
 switchport access vlan 905
 switchport mode access
shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/5
 description TrustLAN
 switchport access vlan 905
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/6
 description FireEye reset port
 switchport access vlan 905
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/7
 description FireEye Span Port VLAN905
!
interface GigabitEthernet1/0/8
 description DMZ-FW01
 switchport access vlan 50
 switchport mode access
 priority-queue out
 spanning-tree portfast
!
interface GigabitEthernet1/0/9
 description GuestWLAN
 switchport access vlan 903
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/10
 description GuestWLAN
 switchport access vlan 903
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/11
 description U/L to wlc-01
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 5 mode on
!
interface GigabitEthernet1/0/12
 description U/L to wlc-01
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 shutdown
 channel-group 5 mode on
!
interface GigabitEthernet1/0/13
 description 906 SSL-Int
 switchport access vlan 906
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/14
 description 906 SSL-Int
 switchport access vlan 906
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/15
 description 906 SSL-Int
 switchport access vlan 906
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/16
 shutdown
!
interface GigabitEthernet1/0/17
 description 907 SSL-Ext
 switchport access vlan 907
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/18
 description 907 SSL-Ext
 switchport access vlan 907
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/19
 description 907 SSL-Ext
 switchport access vlan 907
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/20
 description xxxxxx U/L to 909 Untrust
 switchport access vlan 909
 switchport mode access
 speed 1000
 duplex full
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/21
 description 909 Untrust
 switchport access vlan 909
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/22
 description 909 Untrust
 switchport access vlan 909
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/23
 description 910 QA Ext
 switchport access vlan 910
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/24
 description 909 Untrust
 switchport access vlan 909
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/25
 description Internal
 switchport access vlan 915
 switchport trunk encapsulation dot1q
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/26
 description Internal
 switchport access vlan 915
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/27
 description Internal
 switchport access vlan 916
 switchport trunk encapsulation dot1q
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/28
 description Internal
 switchport access vlan 916
 switchport trunk encapsulation dot1q
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/29
 description FireEye management interface
 switchport access vlan 11
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/30
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/31
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/32
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/33
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/34
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/35
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/36
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/37
 description blade-c1-b3-vmnic2
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/38
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/39
 description Internal
 switchport access vlan 11
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/40
 description Internal
 switchport access vlan 11
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/41
 description blade-c1-b1
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/42
 description blade-c1-b2
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/43
 description blade-c1-b3
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/44
 description blade-c1-b4
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/45
 description U/L to sw02-2f G1/0/49
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 21 mode active
!
interface GigabitEthernet1/0/46
 description blade-c1-b6
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/47
 description Internal
 switchport access vlan 11
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/0/48
 description Internal
 switchport access vlan 11
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet1/1/1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 19 mode active
!
interface GigabitEthernet1/1/2
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 20 mode active
!
interface GigabitEthernet1/1/3
 description U/L to sw01-1f G1/1/1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 22 mode active
!
interface GigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/1
!
interface TenGigabitEthernet1/1/2
!
interface GigabitEthernet2/0/1
 description TrustLAN
 switchport access vlan 905
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/2
 description TrustLAN
 switchport access vlan 905
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/3
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/4
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/5
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/6
 description TrustLAN
 switchport access vlan 905
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/7
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/8
 description DMZ-FW02
 switchport access vlan 50
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/9
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/10
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/11
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/12
 description U/L to wlc-01
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 5 mode on
!
interface GigabitEthernet2/0/13
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/14
 description 906 SSL-Int
 switchport access vlan 906
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/15
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/16
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/17
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/18
 description 907 SSL-Ext
 switchport access vlan 907
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/19
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/20
 description xxxxxx U/L to 909 Untrust
 switchport access vlan 909
 switchport mode access
 shutdown
 speed 1000
 duplex full
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/21
 description 909 Untrust
 switchport access vlan 909
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/22
 description 909 Untrust
 switchport access vlan 909
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/23
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/24
 description 910 QA Ext
 switchport access vlan 910
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/25
 description Internal
 switchport access vlan 915
 switchport trunk encapsulation dot1q
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/26
 description Internal
 switchport access vlan 915
 switchport trunk encapsulation dot1q
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/27
 description Internal
 switchport access vlan 916
 switchport trunk encapsulation dot1q
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/28
 description Internal
 switchport access vlan 916
 switchport trunk encapsulation dot1q
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/29
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/30
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/31
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/32
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/33
 description Internal
 switchport access vlan 915
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/34
 description Internal
 switchport access vlan 916
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/35
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/36
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/37
 description blade-c1-b3-vmnic4
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/38
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/39
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/40
 description Internal
 switchport access vlan 11
 switchport mode access
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/41
 description blade-c1-b1
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/42
 description blade-c1-b2
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/43
 description blade-c1-b3
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/44
 description blade-c1-b4
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/45
 description U/L to sw02-2f G3/0/49
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 21 mode active
!
interface GigabitEthernet2/0/46
 description blade-c1-b6
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 shutdown
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/47
 description Internal
 switchport access vlan 11
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/0/48
 description Internal
 switchport access vlan 11
 switchport mode access
 priority-queue out
 spanning-tree portfast
 spanning-tree guard root
!
interface GigabitEthernet2/1/1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 19 mode active
!
interface GigabitEthernet2/1/2
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 20 mode active
!
interface GigabitEthernet2/1/3
 description U/L to sw01-1f G3/1/1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 901
 switchport mode trunk
 switchport nonegotiate
 channel-group 22 mode active
!
interface GigabitEthernet2/1/4
!
interface TenGigabitEthernet2/1/1
!
interface TenGigabitEthernet2/1/2
!
interface Vlan1
 no ip address
!
interface Vlan11
 ip address 10.16.72.1 255.255.252.0
 ip helper-address 10.16.72.6
 ip helper-address 10.16.72.7
!
interface Vlan12
 ip address 10.16.84.1 255.255.255.0
 ip helper-address 10.16.84.6
!
interface Vlan21
 ip address 10.16.76.1 255.255.255.0
 ip helper-address 10.16.72.6
 ip helper-address 10.16.72.7
!
interface Vlan903
 ip address 10.16.78.1 255.255.255.0
 ip helper-address 10.16.72.6
 ip helper-address 10.16.72.7
!
interface Vlan904
 ip address 10.16.80.1 255.255.252.0
 ip helper-address 10.16.72.6
 ip helper-address 10.16.72.7
!
interface Vlan905
 ip address 192.168.151.2 255.255.255.0
!
interface Vlan909
 no ip address
!
interface Vlan911
 ip address 10.16.77.1 255.255.255.0
 ip helper-address 10.16.72.6
 ip helper-address 10.16.72.7
!
interface Vlan930
 ip address 10.16.85.1 255.255.255.0
 ip helper-address 10.16.84.6
!
ip http server
ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 192.168.151.1
ip route 192.168.240.0 255.255.255.240 10.16.72.94
!
logging esm config
logging trap debugging
logging 10.16.72.90
snmp-server community public RO
snmp-server contact it@xxxxx.com
!
!radius-server host 10.16.72.6 key <removed>
!radius-server host 10.16.72.6 auth-port 1812 acct-port 1813 key <removed>
!
line con 0
line vty 0 4
 exec-timeout 0 0
 authorization exec userAuthorization
 logging synchronous
 login authentication userAuthentication
 transport input ssh
 transport output ssh
line vty 5 15
 exec-timeout 0 0
 authorization exec userAuthorization
 logging synchronous
 login authentication userAuthentication
 transport input ssh
 transport output ssh
!
ntp server 10.16.72.7
end

[Leo Laohoo]

Use IOS version 12.2(55)SE9 or 15.0(2)SE4.

[ranraju · Accepted solution]

You must just focus on why ip input process utilization is consistently around 20%.

Also, there will be no impact on the performance of the switch due to Hulc LED Process.

------

To check what traffic is hitting the CPU of the switch causing IP input process to be at 20%, you need to perform the following.

1. Identify the cpu-queue in which traffic is received. 

> use show controllers cpu-interface (run this multiple times and identify the cpu-queue in which the traffic received is more).

2. Perform a debug of that queue. (it should be safe for you to run this debug, will not cause the switch to go down).

> debug platform cpu-queues <corresponding cpu-queue>

> you can turn of console logging, and enable "logging buffer debugging" (also apply "no terminal mon")

3. check the logs on the switch, it should display packets hitting the CPU, and you must check if they are genuine, or if they shouldn't be making it to the CPU, and get switched in hardware.

You don't need to change the SDM template, as the utilization of the TCAM is well within limits.

Hope this helps..

Ranganath

[Bruce Reed · Accepted solution]

Thanks, here is what show controllers cpu-interface showed (current time):

cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray

----------------- ---------- ---------- ---------- ---------- ----------
rpc               5906891    0          0          0          0
stp               3594340    0          0          0          0
ipc               407765     0          0          0          0
routing protocol  41554410   0          0          0          0
L2 protocol       368661     0          0          0          0
remote console    0          0          0          0          0
sw forwarding     10839      0          0          0          0
host              5731816    0          0          0          0
broadcast         106712160  0          0          0          0
cbt-to-spt        0          0          0          0          0
igmp snooping     40009072   0          0          0          0
icmp              728        0          0          0          0
logging           0          0          0          0          0
rpf-fail          0          0          0          0          0
dstats            0          0          0          0          0
cpu heartbeat     6141884    0          0          0          0

The broadcast queue looked rather large and was growing faster than other queues. I turned debug on for the broadcast queue and saw a lot of broadcast traffic from one of our domain controllers on vlan 11. I also saw a lot of LLMNR traffic destined for 224.0.0.252. The DC broadcast was related to Netbios discovery. I disabled Netbios on both of our DCs and that pruned a lot of broadcast traffic. My CPU situation now looks like:

core01-2f#show proc cpu sort
CPU utilization for five seconds: 17%/0%; one minute: 17%; five minutes: 17%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  85    12153468     2308920       5263  3.35%  3.29%  3.25%   0 RedEarth Tx Mana
  84     9112436     3443940       2645  3.03%  2.47%  2.38%   0 RedEarth I2C dri
 129     6562878      661746       9917  1.59%  1.77%  1.77%   0 hpm counter proc
  12      836312     1063215        786  0.95%  0.42%  0.39%   0 ARP Input
 214   101529580    24126163       4208  0.79%  0.40%  0.34%   0 IP Input
 232     6832348     4687745       1457  0.47%  0.53%  0.67%   0 Spanning Tree
 213      278813     7858444         35  0.31%  0.06%  0.01%   0 IP ARP Retry Age
 320      242195     9331079         25  0.31%  0.03%  0.00%   0 MMON MENG
 181     1320038       62679      21060  0.31%  0.35%  0.32%   0 HQM Stack Proces
 169    31950878     7613079       4196  0.31%  0.75%  0.91%   0 Hulc LED Process
 371       23407      135719        172  0.15%  0.01%  0.00%   0 LACP Protocol
 130      209804      598282        350  0.15%  0.06%  0.04%   0 HRPC pm-counters
  13       15627      324155         48  0.00%  0.00%  0.00%   0 ARP Background
  14           0           1          0  0.00%  0.00%  0.00%   0 CEF MIB API
  15           0           1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
   7           0           2          0  0.00%  0.00%  0.00%   0 Timers

IP input CPU time is way down, so it seems the broadcast traffic from the DCs alone were a large contributor to the problem. What would the core be doing with the broadcast traffic that bogs it down in this way. Here's one of the debug packets from a broadcast within vlan 11:

May 29 21:20:45.895: L2B-Q:Queued L3If: Local Port Fwding L3If:Vlan11 L2If:GigabitEthernet2/1/3 DI:0x703, LT:7, Vlan:11   SrcGPN:680, SrcGID:680, ACLLogIdx:0x0, MacDA:ffff.ffff.ffff, MacSA: a820.664a.f88b   IP_SA:10.16.74.78 IP_DA:10.16.75.255 IP_Proto:17
   TPFFD:E04102A8_000B000B_00A000AB-00000703_1C410000_00000000

[ranraju]

Hi

glad to see you were able to find the cause of the issue.

On higher end platforms you will be allowed to use CoPP to limit the number of packets hitting the CPU, and you can protect your network from broadcast, by implementing broadcast/multicast storm control.

Hope this answers your queries on this post.

Regards,

Ranganath

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

What would the core be doing with the broadcast traffic that bogs it down in this way.

The core switch processes (likely to just discard) those received broadcast packets as a host, i.e. impacting the main CPU.

Broadcasts are often what limits the size of "flat" networks.  Every host gets them, and every host needs to examine them for their relevancy.

Eliminating the source of unnecessary broadcasts was the ideal solution.

Ranganath also mentions CoPP.  It, and/or broadcast storm control, might mitigate the impact of broadcasts, but also keep in mind both "police" broadcasts and so might also drop a broadcast packet you want the switch to "see" and process.  For example, a host ARPing for the GW's MAC.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I agree with both Leo (who suggested "solid" IOS version, especially 55SE9) and Ranganath, who notes you do want to try to resolve the high IP Input CPU usage.

Remember, a 3750 should be forwarding most frames/packets in hardware, so overall CPU utilization has little impact to that, but IP Input is software forwarding, which is much slower and much more capacity limited.

There is a need, of course, for CPU for some control plane services, but as (I believe) different CPU processes have priorities, high CPU utilization caused by something like HULC should have almost no impact against higher priority processes.  I.e. even if HULC drove CPU to 100%, if something like IP Input or routing processes have priority, they will be little impacted.