High CPU Interrupt when using PBR on 3560G

[ali.mouslmani]

Hello,

We have a PBR applied for a certain vlan to redirect  users internet traffic through a dedicated link. The access-list has 5 deny statements with permit any any at the end, so that i direct all traffic through deefault route unless going to our internal servers. The users reported very low bandwidth for the traffic incoming/outgoing to our internal servers. (~30 MB/s on a VLAN with no PBR to a 4MB/s on the VLAN where the policy is applied; around 10x lower)

When this policy is applied on the VLAN, the CPU  usage percantage rises from 7%/0% to 45%/38%. The traffic matching in the access-list for this route-map is processed switched and affecting the switch's performance.

The datasheet for the switch does not contain sizing specifically for process switched traffic, but only the switch fabric rate of 32 Gbps.

interface Vlan30
ip address 10.4.30.1 255.255.255.0
ip helper-address 10.4.2.16
ip policy route-map redirect
standby 30 ip 10.4.30.3
standby 30 priority 110
standby 30 preempt
standby 30 authentication Villa

ip access-list extended redirect
deny ip 10.4.30.0 0.0.0.255 192.168.50.0 0.0.0.255
deny ip 10.4.30.0 0.0.0.255 192.168.55.0 0.0.0.255
deny ip 10.4.30.0 0.0.0.255 10.4.0.0 0.0.255.255
deny ip 10.4.30.0 0.0.0.255 10.1.0.0 0.0.255.255
deny ip 10.4.30.0 0.0.0.255 172.16.1.0 0.0.0.255
permit ip any any

route-map redirect permit 10
match ip address redirect
set ip next-hop 192.168.193.1

Note: the next-hop configured is the core switch(same platform) connected to our servers.

[Mark Malone]

Hey that traffic should be cef switched , are you certain its being process switched.

what process do you see in show proc cpu sorted spiking

[ali.mouslmani]

Hello,

The traffic should be cef switched as you said, but whenever i apply the PBR on the Vlan interface, CPU utilization rises from ~7% to ~45%, with CPU interrupts rising from 0% to 38%, which is abnormal and indicate that interrupts are eating CPU usage. This means that the traffic denied in the PBR to route through its defaullt gateway is interrupting normal CPU operation, and that's why it is processed switched.

[Jon Marshall]

This is a known issue with 3560/3750 switches and is mentioned in the configuration guides.

Using deny lines in your acl causes all matching traffic to be process switched which is why your CPU has gone up so much.

What you really need is the "set ip default next-hop ..." command which tells the switch to use the IP routing table first ie. for traffic between internal subnets but ignore a default route in the routing table so you can then set the next hop IP.

But that command is not supported on your switch.

What you could try is to use two entries in your route map ie. the first entry uses permit lines for all your current deny lines but you don't set a next hop IP and then the second entry just has "permit ip any any" and sets the next hop.

However I'm not sure even this will help as others have reported it made no difference.

Might be worth a try though.

Jon

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I recall reading for some devices, some PBR statements force software forwarding.  That might be the case here, and if so, the 3560/3750 does not have a very fast CPU for software forwarding of packets.