High CPU Utilization 7600-RSP720 and GRE Tunnel

[alibashivan]

Hi,

We've recently replaced a 7200NPE-G2 with 7600-RSP720. We had around 1Gbps GRE tunnels terminated on the previous box with 85% CPU utilization comming from around 30 tunnel interfaces. Also running OSPF and some route-maps. Everyting was doing good. For scaling our connectivity scenario we decided to replace it with a newer and more powerful platform and 7600-RSP720 was our final choice.

After replacing the old 7200 with the new 7600 we surprisingly noticed high cpu utilization (95%) which all came from interrupts and not any specific process. I tried to use some mls configuration commands but there was no success. For the test scenario we've been using GE ports on the RSP720 and not any other line card interfaces.

I double checked the Cisco 7600 IOS guide document, I'm using commands which is supported by PFC3/DFC3 for hardware switching, which is "tunnel source", "tunnel destination" and "tunnel more gre". the only strange paragraph which i suspect is related to this problem is but it's never been detailed anywhere for the correct implementation, i tried different ip addresses for source tunnels but no difference

"Each hardware-assisted tunnel must have a unique source. Hardware-assisted tunnels cannot share a source even if the destinations are different. Use secondary addresses on loopback interfaces or create multiple loopback interfaces. Failure to use unique source addresses may result in control plane failures during software path congestion."

checking "show ip cef not-cef-switched":

IPv4 CEF Packets passed on to next switching layer
Slot  No_adj No_encap Unsupp'ted Redirect  Receive  Options   Access     Frag
RP         0       0     3919465        0   195682        0   331065        0
5/0        0       0           0        0        0        0        0        0
1/0        0       0           0        0        0        0        0        0

also "show ip cef switching statistics"

       Reason                          Drop       Punt  Punt2Host
RP LES Packet destined for us             0     136317      59304
RP LES Encapsulation resource             0    1944707          0
RP LES No adjacency                       8          0          0
RP LES TTL expired                        0          0    1643028
RP LES Fragmentation failed           78770          0          0
RP LES Features                    12836183          0     330963
RP LES Neighbor resolution req            5          2          0
RP LES Total                       12914966    2081026    2033295

All    Total                       12914966    2081026    2033295

Configuration:

interface Tunnel51
description xxxxxxx
bandwidth 1024
ip unnumbered GigabitEthernet5/2.19
ip mtu 1476
ip flow ingress
ip policy route-map redirect
ip ospf cost 100
ip ospf 1 area 1
keepalive 3 3
cdp enable
tunnel source  yy.yy.yy.yy
tunnel destination xx.xx.xx.xx

!

mls flow ip interface-full
mls rp ip route-map

mls ip multicast flow-stat-timer 9

no mls acl tcam share-global
mls cef error action reset

------------------------------------------------------

I also tried the configuration without route-map but still the same.

Anyone had such experince with this problem before?

Thanks in advance

Ali

[francisco_1]

My unsderstanding is On both SRP 720-3C & RSP 720-3CXL handles GRE in hardware. Are packets being fragmented before GRE encapsulation? I think Packets to be fragmented before GRE encapsulation are punted to the Route Processor.

Can you post sh processes cpu | e 0.00%

Also IOS Version

Francisco.

[alibashivan]

I've moved all the traffic back to previous 7200-NPEG2 except two tunnels for the sake of test only. Total bandwith currently in GRE is around 100mbps that's the output of show proc cpu :

CPU utilization for five seconds: 28%/28%; one minute: 29%; five minutes: 30%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
188      467528   4869808         96  0.07%  0.16%  0.16%   0 IP Input

IOS version :

Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-IPSERVICESK9-M), Version 12.2(33)SRC3, RELEASE SOFTWARE (fc2)

"bootdisk:c7600rsp72043-ipservicesk9-mz.122-33.SRC3.bin"

about the fragmentation we have mss-adjust so there won't be much fragmentation, for more saftly I also used "mls cef tunnel fragment".

Regards,

[Giuseppe Larosa · Accepted solution]

Hello Alibashivan,

>> "Each hardware-assisted tunnel must have a unique source. Hardware-assisted tunnels cannot share a source even if the destinations are different. Use secondary addresses on loopback interfaces or create multiple loopback interfaces. Failure to use unique source addresses may result in control plane failures during software path congestion."

are your GRE tunnels using different tunnel source ip addresses or they share the same source?

this can make quite a difference

Hope to help

Giuseppe

[alibashivan]

Dear Giuseppe,

I have 2 tunnels at that moment, I changed the source IP addrsses and it actually fix the problem. Prior to that I had tested it while I had so many tunnles and couldn't see the quick impact on the router CPU but now with just 2 tunnels and less traffic I could clearly see the result. Thanks for the help on the subject, 35% CPU usage is not 3% and I'm going to move more tunnels to the new box and see how it will handle the traffic.

Meanwhile, taking this approach going to consume so many IP addresses as each tunnel will need a unique IP address. Is there any work around on this? like using different loopbacks which all unnumbered to the same IP interface? Or is this the matter of different source IP only?

Regards,

Ali

[Giuseppe Larosa]

Hello Alì,

>> Meanwhile, taking this approach going to consume so many IP addresses as each tunnel will need a unique IP address.

if you can use private IP addresses taken from RFC1918 it shouldn't be a problem

Hope to help

Giuseppe

[belovell · Accepted solution]

Meanwhile, taking this approach going to consume so many IP addresses as each tunnel will need a unique IP address. Is there any work around on this? like using different loopbacks which all unnumbered to the same IP interface? Or is this the matter of different source IP only?
Regards,
Ali

Giuseppe is correct and unfortunately there is no config workaround as it is not a config oddity but a hardware limitation. Each tunnel must have a unquie source IP due to the way the GRE decap path works on the 6K ASICs.

The hardware ASICs on the 6k cannot demux GRE traffic based on both SIP and DIP. It can only be done based on DIP so when GRE packets come into the 6k they must have unique DIP for each tunnel else all incoming traffic in decap path for tunnles that share IP addreses will be punted to the RP for software switching.

-Ben

[alibashivan]

Thnanks for the help and support,

Ali

[DJEphoric]

May be worth noting that with some Crypto Engines you can use the same source address without it affecting the RP.  Example would be the IPSEC-2g or IPSEC-3G SPA engine.