High CPU due to Interrupt on 7301 NPE

[paul amaral]

I have a cisco 7301 (NPE) with 500 megs of ram with a VAM2+ VPN module that is doing about 185Mbs of traffic on a DMVPN tunnel that terminate about 15 spokes. At around 180Mbs I see over 90% cpu usage, looking the show cpu sort there is no process that is using a high percentage of CPU so I know for a fact that the CPU is high because of interrupts. Which unfortunately on this router is not easy to figure out that packets are being handled by interrupts.

I have cef enabled on the router globally and it appears to be working correctly I also have checked the switching paths on the interfaces and those appear to use fast switching and seem normal. I don’t really Know the max throughput for the 7301 is so I’m wondering is the cpu that high solely because of the traffic on the router or is it because im using VPN, I’m under the impression that the VPN module will do ipsec in hardware and not use the CPU or could it be DMVPN that is causing high CPU?

I’m beginning to think its just a matter of having high traffic on the router and its reaching its max throughput, anyone have ideas suggestions I would greatly appreciate it.

Paul

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising  out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The 7301 is about the equivalent of the 7200 with a NPE-1G.

Cisco says:

Performance and Scalability

Cisco 7200 Series and Cisco 7301 routers, combined with the Cisco VAM2+, support up to 280 Mbps of IPSec encryption throughput and up to 5000 IPSec remote-access or site-to-site tunnels.

"Up to" usually means best case.

With tunneled packets, fragmenation will drive up CPU (in later IOS versions, it's in the "fast path" so it will show as interrupt CPU).  Have you done all suggested in: http://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html?

[paul amaral]

Joseph, thanks for your replies, your're  very busy on this forum :).

So did read that doc before and I had no choice but to change the MSS and TCP because of fragmentation issues where users would get borken images on websites or no websites would come up at all on the spoke sites, internet is through hub site. So the issue turned out to be with the tunnel where it needed to fragment and sometimes it couldnt be cause of the DF bit etc. so i had to adjust the MTU/MSS on the tunnels to/from the spoke to the hub. So as i understand it i shouldnt be fragmenting packets but rather telling the selling side to send smaller packets that fit on the tunnel correct? So this shouldnt use cpu.

Also im curious how is fragmentation handled in newer IOS if not at inperrupt ?

here is my tunnel config.

interface Tunnel0
 description Primary DMVPN tunnel
 bandwidth 200000
 ip address 2.2.2.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip flow ingress
 ip nhrp map multicast dynamic
 ip nhrp network-id 1
 ip nhrp holdtime 600
 ip tcp adjust-mss 1360
 ip ospf network broadcast
 ip ospf priority 255
 ip ospf mtu-ignore
 ip ospf 1 area 0
 load-interval 30
 tunnel source GigabitEthernet0/2
 tunnel mode gre multipoint
 tunnel key 1
 tunnel path-mtu-discovery
 tunnel protection ipsec profile generic_ipsec_protection shared

paul

[Vinit Jain]

Hello Paul

Like Joseph mentioned, the fragmentation will be handled as an interrupt in the later IOS version. a quick way to check for fragmentation on the IOS devices is to use the below command:

R1#sh ip traffic | in frag
 0 fragmented, 0 fragments, 0 couldn't fragment
R1#

If the fragmented counter is incrementing, then that would indicate fragmentation is happening. For understanding how the packets are being switched, either being process switched or being switched via fast path, you can use the command show interface <int> switching. This command is useful when investigating high cpu due to interrupts.

Hope this helps.

Thanks

Vinit

[paul amaral]

the router has been up for about 4 weeks and it doesnt seem to be doing much fragmentation. below is the interface stats also which looks good.

I guess the question is with 180Mbs of encrypted traffic using a VAM2+ does it still offload some stuff to the cpu? It interface swithcing path looks good and and there isnt alot of fragments going on at least not for 4 weeks is the router @the max throughput.

aycoast-swansea-ens-gw#sh ip traffic  | in frag
        14594 fragmented, 122994 fragments, 635672 couldn't fragment

GigabitEthernet0/2 to Comcast ENS metro-e network e-lan
          Throttle count          0
                   Drops         RP     389280         SP          0
             SPD Flushes       Fast     642340        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs     551814      Drops          0

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process   92035498 18974746766     294606   35087350
            Cache misses         86          -          -          -
                    Fast 16142265950  757019575 20664354690 25517994608237
               Auton/SSE          0          0          0          0

    Protocol  DEC MOP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0       4284     329868
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       5567     334020       3267     196020
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     376989  118909767     256349   15380940
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising  out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The VAM module should off-load actual encyption.  Without the module, your throughtput might only be a couple of Mbps.  However, the VAM does have its own performance rating, and again, such ratings are often "best case", not always real-world cases.  I.e. your CPU loading, and throughput, might be reasonable.

It can often be difficult to be sure in these cases, because with software based routers, "your mileage may vary".

BTW, I think [?] I recall on the larger 7200s, one/some of the VAMs supported addional modules, per 7200, to increase maximum throughput.

[paul amaral]

thats what I was thinking but wasnt sure. thanks Joseph, Vinit.

paul

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising  out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

"Also im curious how is fragmentation handled in newer IOS if not at inperrupt ?"

The older IOSs, if I remember correctly, would do fragmentation under the IP Input process.

Newer IOSs, do fragmentation under interrupt.

Remember on software based routers, everything is done with the CPU.  Interrupt CPU is processes in the "fast path".

[paul amaral]

I ended up increasing the ipsec replay window size which decreased the CPU spikes i was seeing.

crypto ipsec security-association replay window-size 1024