high CPU on 2900 ISR with ISM VPN and VTI interfaces

[paul amaral]

Hi,

i have an issue i can't quite figure it out, i pretty sure its related to the encryption tunnels im using but im not sure why.

I have a couple of 2911 connected together over a direct ipsec tunnel that is running 70+ mbs of encrypted traffic. Looking at the CPU, im seeing about 90% used due to interrupts. Now i do have an ISM VPN card which from what i understand is supposed to take care of encryption and leaving the main CPU free for other tasks. From what i looked at so far everything looks good, CEF is working fine etc.

My question is why is my CPU so high what can i do to fix the situation. I'm at a loss here as to why the cpu is running so high and im almost 100% is related to the encrypted ipsec tunnel traffic but shouldn't the integrated VPN module be doing all the work. If anyone has some pointers its greatly appreciated.

gw#sh ip cef switching statistics

       Reason                          Drop       Punt  Punt2Host
RP LES No route                          63          0          1
RP LES Packet destined for us             0    1393320      51367
RP LES TTL expired                        0          0     486678
RP LES Fragmentation failed, DF     3014276          0          0
RP LES Discard                        72653          0          0
RP LES Features                        4157          0      31435
RP LES Fragmentation no pak               0          0        764
RP LES Total                        3091149    1393320     570245

All    Total                        3091149    1393320     570245

#sh processes cpu | ex 0.00%__0.00%__0.00%

CPU utilization for five seconds: 90%/89%; one minute: 80%; five minutes: 81%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   2      410484      725275        565  0.00%  0.04%  0.01%   0 Load Meter

   6    15344596      922323      16636  0.00%  0.04%  0.05%   0 Check heaps

  13    25190192       60005     419836  0.00%  0.07%  0.10%   0 Licensing Auto U

  14    15439912     3584474       4307  0.00%  0.08%  0.06%   0 Environmental mo

  67     1064772     3612958        294  0.07%  0.03%  0.02%   0 Per-Second Jobs

  90         612         157       3898  0.00%  0.05%  0.01% 388 Virtual Exec

  96      936256     7203871        129  0.00%  0.03%  0.02%   0 BPSM stat Proces

  98      250836    22404658         11  0.07%  0.00%  0.00%   0 Ethernet Timer C

  99     3526716   345916589         10  0.15%  0.15%  0.15%   0 Ethernet Msec Ti

125      810144    97403729          8  0.00%  0.03%  0.02%   0 IPAM Manager

129      359592     1973843        182  0.00%  0.01%  0.00%   0 IP Input

311     5578200   615440929          9  0.23%  0.27%  0.27%   0 IP SLAs XOS Even

interface Tunnel1
bandwidth 100000
 ip unnumbered GigabitEthernet0/1
 ip nat inside
 ip virtual-reassembly in
 ip tcp adjust-mss 1398
 load-interval 30
 tunnel source GigabitEthernet0/1
 tunnel mode ipsec ipv4
 tunnel destination 172.17.2.5
 tunnel path-mtu-discovery
 tunnel protection ipsec profile ipsec_profile_to_hq
end

Tunnel protocol/transport IPSEC/IP

  Tunnel TTL 255

  Path MTU Discovery, ager 10 mins, min MTU 92

  Tunnel transport MTU 1438 bytes

  Tunnel transmit bandwidth 8000 (kbps)

  Tunnel receive bandwidth 8000 (kbps)

  Tunnel protection via IPSec (profile "VTI-NB")

  Last input never, output never, output hang never

  Last clearing of "show interface" counters 5w2d

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 16072

  Queueing strategy: fifo

  Output queue: 0/0 (size/max)

  30 second input rate 1795000 bits/sec, 3195 packets/sec

  30 second output rate 71052000 bits/sec, 6012 packets/sec

thanks for taking your time,

paul

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The encryption hardware does just that, encryption and decryption; everything else is still left to the main CPU, and there's more to do when supporting encrypted traffic.

In the Cisco's whitepaper I've attached, Table 7 notes a 2911 with 61 Mbps of (aggregated) encrypted traffic at 75% CPU utilization.  So your 90% at 70+ Mbps, doesn't seem way out of line.  (NB: Other features also add to CPU loading, such as your NAT.  See Table 6 for PAT performance.  Also note performance delta between Tables 6 and 7.)

In your config, one interface statement that might help is IP MTU.  You have MTU discovery enabled, which is good, but it will keep discovering that you're unable to send full size (standard) Ethernet packets.  IP MTU sets a new cap, and then MTU discovery only should trigger if an upstream path forces the MTU even smaller.

Even with IP MTU, tcp mss-adjust and path-mtu-discovery, non-TCP traffic won't be impacted by tcp mss-adjust (nor will existing TCP flows that move to the tunnel) and IP and TCP options require more than the usual 40 difference between mss and the packet's size.  Such situations can cause traffic to be fragmented.

So, again, your CPU utilization, for your specific traffic, might be "normal".

Lastly, I recall (?) seeing some posts where virtual-reassembly is unnecessary, but adds to CPU loading.

[paul amaral]

1st off thanks Joseph for the response, I  will read that white paper shortly. Just a quick question what should of set the IP MTU 1438? -  right now the tunnel shows, Tunnel transport MTU 1438 bytes. I didn't hardcode that command because the correct MTU is set by default taking into account the MSS i setup manually of 1398. Do you recommend i set the mss to the cisco recommendation 1360, i really dont want to make things worst and end up with fragments. I will look into the ip reassembly and see if that makes a difference. 

if this is normal ISR ipsec cpu behavior  this kind of sucks, especially now that i just found out the encryption limit is 85kbps which im already hitting at times.

thanks, paul

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

My understanding is, mss-adjust does not set IP MTU, your path-mtu-discovery should have set it.

Cisco's recommendation to set mss-adjust to 1360 (and IP MTU to 1400) are rather conservative.  It's a little more efficient to go larger, but more important is avoiding fragmentation.

Also Cisco's recommendation also assumes MSS is MTU less 40; which it normally is, but again, IP or TCP options will decrease MSS relative to MTU.

Overall, I suspect you're just bumping up against the 2911's maximum performance (that's one of the reasons why there's a 2921 and 2951, and the other "faster" ISRs).

[paul amaral]

Joseph, i agree with you after reading the whitepaper it looks like a router capacity limit issue :(. this whole thing opened my eyes to how underpowered the 2911 for large encryption branches.  im using MSS of 1398 which so far is working great, I took into account 40 bytes for ip/tcp and 62 for ipsec via VTI. I probably should go lower because of options but so far so good. also it looks like when using VTI ipsec tunnel it automatically adjusts the MTU with or without mtu path discovery or adjust-mss. However if i didnt have the adjust-mss i was getting lots of fragment packets because MSS was being adjusted as if the tunnel was 1500 MTU.

thanks, Paul