High CPU utilization - IP Input

[Sundeep Dsouza]

Hello everyone,

Since the time we configured PBR, I am experiencing high CPU utilization on our core switch. I read a Cisco article on how to tackle high CPU utilization problem related to IP Input. I understand that this problem is because of Process switching which is resolved by enabling fast switching or CEF. PBR is configured on VLAN 22. Show ip interface vlan 22 displays that CEF is enabled. However check out the show interface stats and the show interface switching output below:

show interface stats

Vlan22

             Switch path    Pkts In   Chars In   Pkts Out  Chars Out

               Processor  371813672  493604530   54892884 1795894790

             Route cache        126      10306          0          0

                   Total  371813798  493614836   54892884 1795894790

---------------------------------------------------------

show interface switching

Vlan22

          Throttle count          0

        Drops         RP     458480         SP          0

  SPD Flushes       Fast          0        SSE          0

  SPD Aggress       Fast          0

SPD Priority     Inputs          0      Drops          0

     Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out

        Other    Process      53774    3226440          0          0

            Cache misses          0

                    Fast          0          0          0          0

               Auton/SSE          0          0          0          0

           IP    Process  236946062  990475192   53539326 1495552128

            Cache misses          0

                    Fast        126      10306          0          0

               Auton/SSE          0          0          0          0

          ARP    Process  134899587 3800827714    1355357   81321420

            Cache misses          0

                    Fast          0          0          0          0

               Auton/SSE          0          0          0          0

--------------------------------------------------------------------------

Show ip cache verbose

IP routing cache 0 entries, 0 bytes

   0 adds, 0 invalidates, 0 refcounts

Minimum invalidation interval 2 seconds, maximum interval 5 seconds,

   quiet interval 3 seconds, threshold 0 requests

Invalidation rate 0 in last second, 0 in last 3 seconds

Prefix/Length           Age       Interface       Next Hop

--------------------------------------------------------------------------------

It seems from the above output, that the switch is doing process switching and hence the cpu hike. Despite having cef enabled, why does the switch do process switching?

Note: We do have IOS higher than 12.0.

Regards

[Thotsaphon Lueangwattanaphong]

Hi,

   Which switch model and software version are you using? Please post part of PBR configuration.

Toshi

[Sundeep Dsouza]

Cisco WS-C3560G-24TS - C3560-ADVIPSERVICESK9-M), Version 12.2(25)SED1.

--------------------------------------

show route-map User-vlan

route-map XYZ, permit, sequence 10

  Match clauses:

    ip address (access-lists): 189

  Set clauses:

    ip next-hop 192.168.X.X

---------------------------------------

The above route map is applied to vlan 22.

[Edison Ortiz]

Can you provide the output for ACL 189?

If you have deny before permit, this would cause the packets to be processed in software.

[Edison Ortiz]

PBR is processed in hardware unless you are using an unsupported 'match|set' combination.

The only supported hardware assisted features are 'match ip address', 'set ip next-hop' and 'set ip default next-hop'.

Any other option isn't recommended as it will run in software.

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

If not already enabled, you try both flow caching and policy caching.

[Sundeep Dsouza]

CEF is already enabled on the interface, so I guess there is no need for enabling policy caching or flow caching right?

Show cef interface

Vlan22 is up (if_number 2070)

   Corresponding hwidb fast_if_number 2070

   Corresponding hwidb firstsw->if_number 2070

   Internet address is 192.168.22.254/24

   Secondary address 172.20.20.251/24

   Secondary address 192.168.33.2/24

   Secondary address 192.168.792/24

   ICMP redirects are always sent

   IP unicast RPF check is disabled

   Input features: NAT Outside, Policy Routing

   Output features: Post-routing NAT Outside

   Inbound access list is not set

   Outbound access list is not set

   IP policy routing is enabled

   IP policy route map is User-vlan

   BGP based policy accounting on input is disabled

   BGP based policy accounting on output is disabled

   Hardware idb is Vlan22

   Fast switching type 1, interface type 142

   IP CEF switching enabled

   IP CEF switching turbo vector

   IP prefix lookup IPv4 mtrie 8-8-8-8 optimized

   Input fast flags 0x42, Output fast flags 0x100

   ifindex 2070(2070)

   Slot 0 Slot unit 2 VC -1

   Transmit limit accumulator 0x0 (0x0)

   IP MTU 1500

--------------------------------------------------------------------------------------------

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out of the use or inability to use the posting's information even if  Author has been advised of the possibility of such damage.

Posting

Yes, that's my understanding too, i.e, you shouldn't need policy caching with CEF, however I'm making a chicken soup suggestion, as it shouldn't hurt and might help.  More so true for enabling flow caching since it does, I believe, provide benefit beyond what even CEF does.

Am I reading the CEF interface information correctly that you're also doing NAT?

[Sundeep Dsouza]

Nope I am not doing any NAT for this interface, atleast not on the switch. I dont know why its showing up as if we are doing NAT.

I will try enabling policy and flow caching. Any precaution to take before enabling policy and flow caching?

Regrds

[Joseph W. Doherty]

isclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any  purpose. Information provided is for informational purposes only and  should not be construed as rendering professional advice of any kind.  Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever  (including, without limitation, damages for loss of use, data or  profit) arising  out of the use or inability to use the posting's  information even if Author has been advised of the possibility of such  damage.

Posting

Initially none came to mind.  But, rereading the thread I had overlooked we're working with a 3560G.  Unsure if these caching options are supported, and even if they are, might offer little to no benefit.

L3 switches really need to do their "thing" in hardware.  I've found the original 3560/3750 CPU appears slower than a 2800 ISR.  I.e., again, we want the hardware to forward packets.

So, what you really want to figure out is why the processor is being so involved.  There's likely some Cisco 3560/3750 white papers or tech notes that might help.  Off the top of my mind is whether the right SDM template is being used and/or whether TCAM resources are being exceeded.

[Sundeep Dsouza]

All that I know is that the IP Input process is hogging the CPU. There is a white paper available at Cisco.com on how to tackle this issue. All it says is to enable fast caching or CEF. Since I already have CEF enabled, it leaves me no other option. Hence I need some more insight on what is causing this process to utilize the CPU.

Regards

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind.  Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if  Author has been advised of the possibility of such  damage.

Posting

You've seen documents like?

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807ccc79.shtml#perf

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00801e7bb9.shtml

NB: the 3750 is the "stackable" twin of the 3560

[Sundeep Dsouza]

Thanks Joseph,

I went through the links you posted and found that it is not recommended to use ACLs with deny ACEs while using PBR.

Do not match ACLs with deny ACEs. Packets that match a deny ACE are           sent to the CPU, which can cause high CPU utilization.

In our case, we are using ACLs with deny ACEs and we need to have those deny statements so as to forward those traffic normally.

For you information we are using the routing template(SDM).

Regards