cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
Join Customer Connection to register!
9459
Views
0
Helpful
6
Replies
Highlighted

3750 stack with high cpu (ip input)

Howdy

We added Site02 for redundancy and DR. Also added new SAN and ESX hosts.

The CPU on device Site01Server01 is not very happy when files are being transferred between the sites.

Since the setup is the same at both sites, why is the problem only at site01?

Drawings attached, any help appreciated.

Site02 is newer and we have two 3750X-24 with IOS 12.2(55)SE3 on ipservices, for site02server01; no problem.

Site01server01 is a stack of two 3750G-24PS with IOS 12.2(58)SE on ipservices.

Some output from site01server01 switch:

site01server01#sh proc cpu | e 0.00+.+0.00

CPU utilization for five seconds: 97%/17%; one minute: 90%; five minutes: 62%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   4     6224333      554508      11224  0.00%  0.25%  0.19%   0 Check heaps

  12      203092      453904        447  0.15%  0.01%  0.00%   0 ARP Input

  73     1515032       55326      27383  0.63%  0.07%  0.01%   0 Per-minute Jobs

  74       97220     3278016         29  0.00%  0.05%  0.01%   0 Per-Second Jobs

  76      629199    74578814          8  0.31%  0.22%  0.13%   0 RedEarth Tx Mana

  77      151057   128968304          1  0.00%  0.08%  0.04%   0 RedEarth Rx Mana

  82     2268113   172974758         13  1.43%  0.98%  0.68%   0 Fifo Error Detec

  86      436894     3360808        129  0.15%  0.03%  0.00%   0 hrpc <- response

  96      332508    96520608          3  0.00%  0.15%  0.08%   0 HLFM address lea

  98      162516    96547766          1  0.00%  0.06%  0.03%   0 HLFM address ret

102     6498138      326733      19888  0.00%  0.23%  0.18%   0 HULC Tcam Memory

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

117      691357    40954329         16  0.31%  0.37%  0.22%   0 hpm main process

121     1996393     5241576        380  0.31%  0.40%  0.27%   0 hpm counter proc

122      948094     3853974        246  0.00%  0.04%  0.01%   0 HRPC pm-counters

128       43142     2563878         16  0.15%  0.01%  0.00%   0 hl3mm

147      270764    16236135         16  0.00%  0.10%  0.06%   0 Hulc Storm Contr

161     9288189    97322877         95  4.14%  4.67%  3.13%   0 Hulc LED Process

171     2199514      653502       3365  0.15%  0.15%  0.12%   0 HQM Stack Proces

172     3075305     2610329       1178  0.00%  0.07%  0.08%   0 HRPC qos request

184       80860     3253008         24  0.00%  0.02%  0.01%   0 PI MATM Aging Pr

217      244402     1546665        158  0.00%  0.05%  0.01%   0 CDP Protocol

223      156320    96520712          1  0.15%  0.07%  0.01%   0 IP ARP Retry Age

224   104212895    19557297       5328 65.07% 57.10% 38.66%   0 IP Input

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

238    13304837    98349065        135  0.47%  0.54%  0.39%   0 Spanning Tree

254      122562     5036044         24  0.15%  0.04%  0.00%   0 CEF: IPv4 proces

274      158370     7225103         21  0.00%  0.05%  0.02%   0 Marvell wk-a Pow

289      104174     6501504         16  0.00%  0.06%  0.01%   0 DHCPD Receive

294       83646    32680692          2  0.00%  0.06%  0.01%   0 MDFS RP process

302         942         237       3974  0.15%  0.86%  0.24%   2 SSH Process

305      229805   102322961          2  0.15%  0.10%  0.05%   0 MMON MENG

311     2318417      126503      18326  0.00%  0.41%  0.43%   0 crypto sw pk pro

317       65247    18691657          3  0.15%  0.02%  0.00%   0 PM Callback

318      412392     1063829        387  0.15%  0.01%  0.00%   0 VLAN Manager

357      111523    11221896          9  0.00%  0.08%  0.02%   0 IGMP Input

360      238682    24226160          9  0.63%  0.17%  0.08%   0 PIM Process

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

362     1499632     5631482        266  0.48%  0.16%  0.12%   0 IP SNMP

363      406557     2811398        144  0.32%  0.04%  0.00%   0 PDU DISPATCHER

364     2056115     2886865        712  0.32%  0.18%  0.15%   0 SNMP ENGINE

site01server01#show interfaces switching
Vlan1

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    5961479  597177546    3718046  432556707
            Cache misses          0          -          -          -
                    Fast          6        570          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     433495   26009700       1984     119040
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
Vlan157 iLo Mgmt

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      68632    6044310     218556   18879116
            Cache misses          0          -          -          -
                    Fast          0          0          3        316
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      11900     714000      27623    1657380
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         42       2520          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
Vlan165 Storage Subnet

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       8195    1171838  325747464 152512709543
            Cache misses          0          -          -          -
                    Fast          0          0          5        584
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       8701     522060      10248     614880
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         11        660          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
Vlan2002 InterDataCenter_PtP_Link

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process  325906188 152530627679     302429   23589582
            Cache misses          0          -          -          -
                    Fast          3        291          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         80       4800         81       4860
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.

site01server01#show ip traffic
IP statistics:
  Rcvd:  333774251 total, 4774621 local destination
         0 format errors, 0 checksum errors, 8527 bad hop count
         0 unknown protocol, 1569160 not a gateway
         0 security failures, 0 bad options, 639471 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 639471 alert, 0 cipso, 0 ump
         0 other
  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
         661 fragmented, 0 couldn't fragment
  Bcast: 8383 received, 1 sent
  Mcast: 888759 received, 510860 sent
  Sent:  4363457 generated, 2366166154 forwarded
  Drop:  163 encapsulation failed, 0 unresolved, 0 no adjacency
         0 no route, 0 unicast RPF, 0 forced drop
         0 options denied, 0 source IP address zero

ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 2037 unreachable
        988657 echo, 159 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
  Sent: 0 redirects, 26487 unreachable, 170 echo, 988657 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp
        0 info reply, 6 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
  Rcvd: 23080 total, 0 checksum errors, 314 no port
  Sent: 23562 total

UDP statistics:
  Rcvd: 3407912 total, 0 checksum errors, 1662 no port
  Sent: 2813384 total, 33404 forwarded broadcasts

BGP statistics:
  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh, 0 unrecognized
  Sent: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh

EIGRP-IPv4 statistics:
  Rcvd: 0 total
  Sent: 0 total

PIMv2 statistics: Sent/Received
  Total: 111042/75571, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 111042/75545
  Join/Prunes: 0/26, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  Queue drops: 0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 54620/36609, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 36034/18584, Host Reports: 18586/18024, Host Leaves: 0/1
  DVMRP: 0/0, PIM: 0/0
  Queue drops: 0

OSPF statistics:
  Last clearing of OSPF traffic counters never

  Rcvd: 240568 total, 0 checksum errors
        227568 hello, 14 database desc, 0 link state req
        8731 link state updates, 4255 link state acks

  Sent: 345228 total
        341143 hello, 12 database desc, 6 link state req
        542 link state updates, 3525 link state acks

ARP statistics:
  Rcvd: 452702 requests, 1806 replies, 53 reverse, 0 other
  Sent: 28170 requests, 11845 replies (1 proxy), 0 reverse
  Drop due to input queue full: 0

site01server01#sh ip int vlan2002

Vlan2002 is up, line protocol is up

  Internet address is 192.168.199.66/30

  Broadcast address is 255.255.255.255

  Address determined by setup command

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is enabled

  Outgoing access list is not set

  Inbound  access list is not set

  Proxy ARP is enabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are never sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP Flow switching is disabled

  IP CEF switching is enabled

  IP CEF switching turbo vector

  IP Null turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast, CEF

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Probe proxy name replies are disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: MCI Check

  Output features: Check hwidb

  WCCP Redirect outbound is disabled

  WCCP Redirect inbound is disabled

  WCCP Redirect exclude is disabled

site01server01#sh ip int vlan165

Vlan165 is up, line protocol is up

  Internet address is 192.168.165.1/24

  Broadcast address is 255.255.255.255

  Address determined by setup command

  MTU is 1500 bytes

  Helper addresses are 192.168.1.10

                       192.168.164.79

                       192.168.1.79

                       192.168.235.79

  Directed broadcast forwarding is enabled

  Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13

      224.0.0.5 224.0.0.6

  Outgoing access list is not set

  Inbound  access list is not set

  Proxy ARP is enabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are never sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP Flow switching is disabled

  IP CEF switching is enabled

  IP CEF switching turbo vector

  IP Null turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast, CEF

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Probe proxy name replies are disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: MCI Check

  Output features: Check hwidb

  WCCP Redirect outbound is disabled

  WCCP Redirect inbound is disabled

  WCCP Redirect exclude is disabled

site01server01#sh ip int vlan157

Vlan157 is up, line protocol is up

  Internet address is 192.168.157.1/24

  Broadcast address is 255.255.255.255

  Address determined by setup command

  MTU is 1500 bytes

  Helper addresses are 192.168.1.10

                       192.168.164.79

                       192.168.1.79

                       192.168.235.79

  Directed broadcast forwarding is enabled

  Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13

      224.0.0.5 224.0.0.6

  Outgoing access list is not set

  Inbound  access list is not set

  Proxy ARP is enabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are never sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP Flow switching is disabled

  IP CEF switching is enabled

  IP CEF switching turbo vector

  IP Null turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast, CEF

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Probe proxy name replies are disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: MCI Check

  Output features: Check hwidb

  WCCP Redirect outbound is disabled

  WCCP Redirect inbound is disabled

  WCCP Redirect exclude is disabled site01server01#sh ip int vlan2002
Vlan2002 is up, line protocol is up
  Internet address is 192.168.199.66/30
  Broadcast address is 255.255.255.255
  Address determined by setup command
  MTU is 1500 bytes
  Helper address is not set
  Directed broadcast forwarding is enabled
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is enabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is enabled
  ICMP redirects are never sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP Flow switching is disabled
  IP CEF switching is enabled
  IP CEF switching turbo vector
  IP Null turbo vector
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, CEF
  Router Discovery is disabled
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled
  Probe proxy name replies are disabled
  Policy routing is disabled
  Network address translation is disabled
  BGP Policy Mapping is disabled
  Input features: MCI Check
  Output features: Check hwidb
  WCCP Redirect outbound is disabled
  WCCP Redirect inbound is disabled
  WCCP Redirect exclude is disabled
site01server01#sh ip int vlan165
Vlan165 is up, line protocol is up
  Internet address is 192.168.165.1/24
  Broadcast address is 255.255.255.255
  Address determined by setup command
  MTU is 1500 bytes
  Helper addresses are 192.168.1.10
                       192.168.164.79
                       192.168.1.79
                       192.168.235.79
  Directed broadcast forwarding is enabled
  Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13
      224.0.0.5 224.0.0.6
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is enabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is enabled
  ICMP redirects are never sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP Flow switching is disabled
  IP CEF switching is enabled
  IP CEF switching turbo vector
  IP Null turbo vector
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, CEF
  Router Discovery is disabled
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled
  Probe proxy name replies are disabled
  Policy routing is disabled
  Network address translation is disabled
  BGP Policy Mapping is disabled
  Input features: MCI Check
  Output features: Check hwidb
  WCCP Redirect outbound is disabled
  WCCP Redirect inbound is disabled
  WCCP Redirect exclude is disabled
site01server01#sh ip int vlan157
Vlan157 is up, line protocol is up
  Internet address is 192.168.157.1/24
  Broadcast address is 255.255.255.255
  Address determined by setup command
  MTU is 1500 bytes
  Helper addresses are 192.168.1.10
                       192.168.164.79
                       192.168.1.79
                       192.168.235.79
  Directed broadcast forwarding is enabled
  Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13
      224.0.0.5 224.0.0.6
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is enabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is enabled
  ICMP redirects are never sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP Flow switching is disabled
  IP CEF switching is enabled
  IP CEF switching turbo vector
  IP Null turbo vector
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, CEF
  Router Discovery is disabled
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled
  Probe proxy name replies are disabled
  Policy routing is disabled
  Network address translation is disabled
  BGP Policy Mapping is disabled
  Input features: MCI Check
  Output features: Check hwidb
  WCCP Redirect outbound is disabled
  WCCP Redirect inbound is disabled
  WCCP Redirect exclude is disabled

physical.jpg

Figure 0.1 Physical

logical.jpg

Figure 0.2 Logical

6 REPLIES 6
Highlighted
Beginner

Hi, compare the SDM usage on both stack to see if the one having the problems is full at some feature

show platform tcam utilization

Are you doing PBR on this stack, if so check the restrictions here:

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml#pbr


- Jorge

Highlighted

Hi Jorge

No PBR on the stack, thanks mate.

Highlighted
Hall of Fame Community Legend

Roll-back/downgrade your IOS from 12.2(58)SEx to 12.2(55)SE1 or later.  Starting from 12.2(58)SE and 15.0(1)SE, these two versions have a memory leak problem that Cisco has been trying to fix for months (particularly the former).

Highlighted

Thanks for the heads up.
Memory is behaving, still have plenty to spare.
The high CPU just went away, tested with high traffic loads and CPU just stayed at 6%.
Looks like it just had a cpl of off days?

Highlighted
Cisco Employee

If you are having a high CPU in IP Input it is best to see what exact the traffic is that is being switched

by the CPU.

A first step would be to use the show controller cpu  command to determine the exact queue that increments

a lot when the CPU is high

If you know which queue traffic is on you can then run debug platform cpu

Best to disable console loggin and just turn it on and of again after a few seconds and look with show log

what the exact packets were.

Highlighted
Cisco Employee

hi ,

 

you can implement a safe debug as follow :

 

(config)#no logging console 
(config)#no logging monitor 

(config)#logging buffer 128000

 

after that clear the log

 

#debug ip packet detail

 

to check the trend of the traffic (which IP ,interface ..)

 

Regards

Content for Community-Ad