Fast Switching vs CEF? Need to lower IP Input Proc Utilization.

shiermanm · ‎07-06-2010

Hello,

I have a 2811 w/VPN Module which is frequently exhibiting High CPU utilization due to the "IP Input" process. A bit of digging turned up that this is due to process switched packets however we are already running CEF for this router, and I have verified via "show interfaces switching" that it is being used to some degree.

My big question is whether I could expect any improvement on this front in modifying my configuration to "fast switching" or whether CEF is already going to be handling the per-destination routing with greater efficiency?

Is there any way to diagnose whether we simply need to upgrade?

Big thanks.

gephelps · ‎07-06-2010

CEF switching is the best method so I wouldn't worry about trying to change it.

IP Input is used for process switched traffic as you stated. There are many potential causes for traffic to be process switched. You didn't state which version of IOS you were running. Can you paste in the output from the following commands:

terminal exec prompt timestamp

show cef not-cef-switched

show ip cef switching statistics

show ip cef switching statistics feature

show ip traffic

How much of the CPU is due to the IP Input process?

shiermanm · ‎07-06-2010

PHXRTR02#sho cef not-cef-switched
Load for five secs: 88%/58%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:49:50.700 MST Tue Jul 6 2010

% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP 0 0 124603860 0 2474 0 5008 0

PHXRTR02#sho ip cef switch stat
Load for five secs: 88%/57%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:50:24.060 MST Tue Jul 6 2010

       Reason                          Drop       Punt Punt2Host
RP LES Packet destined for us             0       1267       1207
RP LES Encapsulation resource             0       2824          0
RP LES Incomplete adjacency               0          0      10366
RP LES TTL expired                        0          0       2883
RP LES Features                         434          0 124601258
RP LES Total                            434       4091 124615714

All Total 434 4091 124615714

PHXRTR02#sho ip cef switch stat feat
Load for five secs: 82%/54%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:50:40.813 MST Tue Jul 6 2010

IPv4 CEF input features:
       Feature                Drop    Consume       Punt Punt2Host Gave route
       Virtual Fragment         91          0          0          0          0
       Access List               3          0          0       5008          0
       NAT Outside               0          0          0 124602939          0
Total                           94          0          0 124607947          0

IPv4 CEF output features:
Feature Drop Consume Punt Punt2Host New i/f
Total 0 0 0 0 0

IPv4 CEF post-encap features:
       Feature                Drop    Consume       Punt Punt2Host    New i/f
       IPSEC Post-encap        340 141966553          0          0          0
Total                          340 141966553          0          0          0

IPv4 CEF for us features:
Feature Drop Consume Punt Punt2Host New i/f
Total 0 0 0 0 0

IPv4 CEF punt features:
Feature Drop Consume Punt Punt2Host New i/f
Total 0 0 0 0 0

IPv4 CEF local features:
Feature Drop Consume Punt Punt2Host Gave route
Total 0 0 0 0 0

PHXRTR02#sho ip traffic
Load for five secs: 89%/59%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:50:52.425 MST Tue Jul 6 2010

IP statistics:
Rcvd: 2364200412 total, 1190951521 local destination
         0 format errors, 0 checksum errors, 766 bad hop count
         1 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other
Frags: 718 reassembled, 0 timeouts, 0 couldn't reassemble
         4970151 fragmented, 9940568 fragments, 530 couldn't fragment
Bcast: 366 received, 0 sent
Mcast: 5302283 received, 5267912 sent
Sent: 38786751 generated, 3812043703 forwarded
Drop: 780 encapsulation failed, 0 unresolved, 0 no adjacency
         67 no route, 0 unicast RPF, 0 forced drop
         0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
         0 physical broadcast

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 79304 unreachable
        798634 echo, 1044 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 4 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
        271 time exceeded, 0 info replies
Sent: 0 redirects, 48216 unreachable, 1060 echo, 798634 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 4 timestamp replies
        0 info reply, 760 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
Rcvd: 133304 total, 9 checksum errors, 6260 no port
Sent: 166045 total

UDP statistics:
Rcvd: 12888284 total, 0 checksum errors, 4747 no port
Sent: 12935831 total, 0 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

IP-EIGRP statistics:
Rcvd: 0 total
Sent: 0 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

OSPF statistics:
Rcvd: 5314417 total, 0 checksum errors
4083340 hello, 933 database desc, 60 link state req
1228547 link state updates, 1450 link state acks

Sent: 5288704 total
4115824 hello, 754 database desc, 43 link state req
653251 link state updates, 518832 link state acks

ARP statistics:
Rcvd: 190483 requests, 119 replies, 0 reverse, 0 other
Sent: 1591 requests, 2726 replies (0 proxy), 0 reverse
Drop due to input queue full: 0

shiermanm · ‎07-06-2010

Sorry for the multiple posts, in answer to your last question, here is the additional output for cpu utilization:

PHXRTR02#sho proc cpu sort
Load for five secs: 85%/56%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:53:03.658 MST Tue Jul 6 2010

CPU utilization for five seconds: 85%/56%; one minute: 85%; five minutes: 85%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
119 1853535196 1219057691       1520 27.57% 27.47% 27.63%   0 IP Input
184    16137016   411358600         39 0.55% 0.53% 0.52%   0 HQF Shaper Backg
234        4620        2097       2203 0.39% 0.06% 0.07% 514 SSH Process
37      251044     8403647         29 0.07% 0.01% 0.00%   0 Net Background
41     1567876    19554786         80 0.07% 0.08% 0.07%   0 Per-Second Jobs
185     1129920   194449152          5 0.07% 0.04% 0.05%   0 RBSCP Background
   6       47840       52759        906 0.00% 0.00% 0.00%   0 Pool Manager

Thanks!

shiermanm · ‎07-07-2010

Anyone have any thoughts on this? It's a shameless bump on my end I know ;D

gephelps · ‎07-07-2010

We really still need to collect more info. The difference between the 2 show ip cef switching statistics commands was 50 seconds. In this time the number of punts for features was 6,689 packets. These are sent to be process switched, but they alone may or may not be the problem. IP Input is ony used to process switch traffic. In a similiar fashion the Input queue of each interface is only used for process switched traffic. I would run the following command to filter down the interfaces:

show interfaces | include protocol|Input

In the output you want to take note of input queue with the most packets. You will probably see a lot of Input queue drops on the same interface. Then you can use the following command to see what that traffic is:

show buffers input-interface packet | include source

The output will be a list of all of the packets which are currently in the Input queue at that instant. The source IP, dest IP, source port, dest port, and protocol will be listed. A lot of times you can find a single stream which is the offending stream. The trick is to determine what shouldn't be there, because some of the traffic is likely to be legitamite.

As an example I had a customer today where we ran the above command sever times and nearly all of the traffic was using the port for SSH. All of the packets came from the same source and were destined to the same destination. The destination IP address wasn't configured on the router so I then checked the routing table. We didn't have a route to the destination which is why the traffic was being punted. If you can't find something glaring you can take the full output form the show buffers command and we can start decoding packets to look for other potential causes.

shiermanm · ‎07-07-2010

Hi George,

The queue is mostly on our FA0/1(OUTSIDE-Internet) interface however there is a smaller queue on the TU3 interface. When I ran the commands you provided it revealed:

PHXRTR02#sho buffers input-interface fastEthernet 0/1 packet | include source
source: <PHXRTR02-Outside(FA0/1) IP>, destination: <CGYRTR01-Outside IP>, id: 0x7BCF, ttl: 239, prot: 50

I am not sure about the port in use but showing IP cache flow did give me SrcP in HEX of 32E1(13025 DEC) and DstP in HEX of FE54(65108 DEC):

PHXRTR02#sho ip cache flow | inc <PHX-Outside-IP>

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP Pkts
Fa0/1         <CGY-Outside-IP>   Local         <PHX-Outside-IP> 32 32E1 FE54    14K
Fa0/1         <CGY-Outside-IP>   Local         <PHX-Outside-IP> 32 32E1 FE54    10K

Again, it appears to be on port 13025 but I am not entirely sure what this is used for but it might be this is the encrypted traffic for the VPN between the two sites on TU3?

gephelps · ‎07-07-2010

That is all encrypted traffic. Protocol 50 is for ESP. In the show ip cache flow output the Pr column is protocol and it is reported in hex. 0x32 == 50 in decimal. When you use the show buffers command this only displays packets at that given instant. You may need to run it several times to get a good sampling of packets. From the output it is hard to determine how many packets you looked at.

shiermanm · ‎07-08-2010

Hmm, isn't the packet count in the ip-cache flow indicitive of the amount of ESP traffic is flowing? It also says in the command that active flows timeout in 30 minutes. If that is the case, here is the current flow, but please do note that overall traffic is much lower right now, and CPU utilization as well.

SrcIf            SrcIPaddress         DstIf              DstIPaddress         Pr      SrcP      DstP       Pkts
Fa0/1           Local                32      16F5      F91F        148K
Fa0/1           Local                32      16F5      F91F        223K

As you suggested I ran the command several times yesterday and today and never saw anything but ESP packets in the buffers. I guess the other thing I am confused about is that this flow is showing up in the "ip cache flow" command, does that mean that it is being properly CEF switched?