Strange packet drops

[OOO Complex Telesystems OOO CTS]

Hello

Pls give me some advice in troubleshooting.

We have several sites, connected with one ISP via L3 mpls VPNs. There is no routing protocol between our routers and ISP routers, we have p2p GRE tunnels from each site to each other site with OSPF inside them. One site has just static routing inside the GRE.

Now we have the following strange situation:

Ping from site1 router to local ISP router is clean. Ping from site one to the remote ISP router is also clean. Ping from site1 router to site2 router is not clean, we are getting 5% drops. Ping from site2 router to its local ISP router is also clean.

I have no clue how to deal with it. It seems that our routers are dropping ICMP but the channels are not overused, there are no rules to limit ICMP, the CPU load is about 5-7%. Drops appear both when packets travel inside the tunnel and outside the tunnel.

ISP says that it can successfully ping our interfaces from any point of their network.

We have 3845 routers at our sites, ios versions are different - 12.4(7d)advipservices, 12.4(24)T1advipservices.

Traceroutes between this sites are identical. We use  NM-16ESW module interfaces for this WAN channels.

interface configuration:

site 1

interface FastEthernet2/7
no switchport
ip address x.x.x.x x.x.x.x
ip flow ingress
load-interval 30
duplex full
speed 10
no cdp enable
end

interface Tunnel266
bandwidth 2048
ip unnumbered Loopback0
ip mtu 1476
ip flow ingress
ip tcp adjust-mss 1436
load-interval 30
qos pre-classify
keepalive 2 3
cdp enable
tunnel source FastEthernet2/7
tunnel destination y.y.y.y y.y.y.y.y

site 2

interface FastEthernet2/0
no switchport
ip address y.y.y.y y.y.y.y.y
ip flow ingress
ip flow egress
duplex full
speed 10
no cdp enable

interface Tunnel259
ip unnumbered Loopback0
ip mtu 1476
ip flow ingress
ip tcp adjust-mss 1436
load-interval 30
qos pre-classify
keepalive 2 3
cdp enable
tunnel source FastEthernet2/0
tunnel destination x.x.x.x x.x.x.x

Maybe someone had the same expirience. Are there any ideas how to troubleshoot it?

Thanks

[OOO Complex Telesystems OOO CTS]

We need several gre tunnels to interconnect sites one with each other. DMVPN would be a better choice here but we are not ready to implement it right now.

Tunnels from the site have the same source (interface that points to the ISP) and different destination(interface that points to the isp at the remote site). This tunnels require the same source to come up because ISP routes only border networks.

According to the docs receive counter increases when GRE tunnel terminates on the router. GRE encapsulated packet is not cef switched, and decapsulated packet is cef switched. So perhaps it is normal for this counter to increase.

What is similar on all sites is that show cef drop command shows a lot drops with usnupported reason. It is increasing rapidly, about 50 packets per second and it seems to me that it is not a pure packet drop but sending a packet to another layer of processing(sending to process-swithcing).

On newer IOSes sh ip cef switching statistics command shows that there are a lot of punt and punt to host packets, increasing rapidly.

There is no pbr or nat on the routers and i have removed netflow configurtion but still sh not cef switched command show increasing unsupported counters.

[milan.kulik]

Hi,

my understanding is quite similar.

What about trying show adjacency [interface-type interface-number] internal

http://www.cisco.com/en/US/tech/tk827/tk831/technologies_tech_note09186a0080094303.shtml

the punt adjacencies might be displayed.

Or debug ip cef receive

http://www.cisco.com/en/US/docs/ios/12_3/debug/command/reference/dbg_h1g.html#wp1086026

which might show the source of punted packets?

BR,

Milan