Process switched packets - CEF & input-queue

[diondohmen]

Hi there,

I am trying to find the reason for the high number of input-queue drops on a 2821:

  Input queue: 8/150/3790/0 (size/max/drops/flushes); Total output drops: 0

The interface concerned has CEF enabled:

IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP CEF switching is enabled

Reading through a few docs online I have made the following assumptions;

- the reason for the input-queue to get filled are packets for which no valid cached adj is found in the CEF cache (or a CEF unsupported feature)

- the packets within the input-queue are getting process-switched

- to find out which packets are filled within the input queue: show buffers input-interface g0/1 header

this gives us a list of UDP p161 packets (snmpget's)

- I can't find a possible reason for these UDP packets (p161) not getting CEF switched (interrupt switched) because I can find a match (valid adj) within the cache?

- running "show cef not-cef-switched" shows me these packets fall into the category: "Receive" 

CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP         0         0                   0              0         6972         0           0        0

Cisco Docs mention these "receive" packets are packets destined for the router itself which I can't match...

Running "debug ip cef receive" gives us the following output:

407683: Dec 14 14:55:34.323 cet: IP-CEF: Receive packet for 145.7.25.198 (process switch)
407684: Dec 14 14:55:34.323 cet: IP-CEF: Receive packet for 145.7.25.198 (process switch)
407685: Dec 14 14:55:34.331 cet: CEF-receive: Receive packet for 145.7.25.198
407686: Dec 14 14:55:34.331 cet: CEF-Receive: Packet for 145.7.25.198 -- receive
407687: Dec 14 14:55:34.335 cet: IP-CEF: Receive packet for 145.7.25.198 (process switch)
407688: Dec 14 14:55:34.339 cet: IP-CEF: Receive packet for 145.7.25.198 (process switch)
407689: Dec 14 14:55:34.923 cet: CEF-receive: Receive packet for 145.7.25.198

As you can see, these actually are packets which get processed switched. Now I need to know what kind of packets these are?

I have already tried to do a "debug ip packet detail ACL" to see if these are packets pointed to the router itself, but the number of packets I see from the debug are much less than the number of lines which get spit out of "debug ip cef receive" 

I hope someone can give some poiters to get more insight in these processed switched packets....

thanks!

[Joseph W. Doherty · Accepted solution]

If the issue is unnecessary process switching, I recall (?) there's a troubleshooting paper on Cisco's main site.

However, it also just might be a performance limitation of the 2821 . . .

Is the interface in question gig Ethernet, running at gig?  If so, the 2821 cannot usually support a large gig burst.

You can try increasing the input queue depth (even to maximum) and/or slow ingress (e.g. run interface at 100 Mbps) to avoid gig ingress bursts losing packets due to the CPU being unable to keep up.

[diondohmen]

Hi Joseph,

The if is running at 1000/full (fixed).

I would like to know if my troubleshooting steps are correct and what kind of packets are meant by "Receive" under "show cef not-cef-switched". And when I have a look at the buffer, why are these UDP snmp packets not CEF switched? This would mean that routing at this particular router could be way more efficient?

I can indeed increase the hold-queue, but then I would still have these SNMP packets processed switched...

I thought the limit of the hold-queue is 4096... is this safe on a 2821?

[Joseph W. Doherty]

Without doing some research, cannot comment on your troubleshooting approach.

Unable to say with 100% assurance, that there's no issue increasing hold queue to max, but recall a whitepaper, for 7200s, saying it was generally okay, as you're just trying to give the CPU a chance to keep up with an ingress burst and you cannot as easily select what traffic to drop (although if the router offers SPD, you have some minimal control).

[diondohmen]

The thing I've not mentioned before, the interfaces which handle these packets actually have "ip nat inside"  en "ip nat outside" configured. I have not find any info about NAT & CEF but could it be the case that when using NAT (including route-maps within "ip nat inside source route-map") CEF is bypassed because the router will punt all of this traffic to the CPU?