MGRE strange behaviour

[Jonn cos]

Hi all experts.

For quite some time i am facing this strange issue. I will explain it but before let me highlight the IOS in which i have experienced this behaviour

1) c3845-advsecurityk9-mz.124-22.T.bin

2) c3845-adventerprisek9-mz.124-22.T2.bin

3) c3845-advipservicesk9-mz.124-20.T.bin

4) c3845-advsecurityk9-mz.124-20.T.bin

The issue is

1) After reloading the router

2) Or suddenly while router is on

My MGre tunnels (without tunnel protection) gets stuck. When i do sh ip os neighbor tunnel 111, all the adjacencies are shown in INIT state. Now i have tried the following

1) Shut and no shut the tunnel interface, no use

2) Delete the tunnel and repaste it, no use

3) Delete the tunnel and repaste it with a different tunnel number, lets say (in above case)

    no int tun 111

    int tun 222

     <config>

And everything things starts working again !!! Can someone help me why is this happening ? seems like a bug to me, but i dont know which IOS to use now

[Peter Paluch]

Jonn,

Interesting issue indeed. Does this happen on a hub or on a spoke router? Can you paste the complete configuration of your Tunnel interface here (both spoke and hub)? Also it would be interesting to see the output of the following debugs:

debug ip ospf hello

debug ip ospf adj

on both sides of the mGRE tunnel (take only two routers - hub and a selected spoke, some of them stuck in the INIT state).

An adjacency stuck in the INIT state means that our router is receiving and understanding the OSPF Hello packets from the neighbor but the neighbor is not seeing our router. Thus it looks like the mGRE tunnel becomes "unidirectional" for whatever reason (don't take me at the word right now, I'm just describing my first impression). That could have something to do with multicast mappings.

Best regards,

Peter

[Jonn cos]

Dear Peter,

Thanks for taking interest in this post.

Here is the configuration for HUB

interface Tunnel1101
description DMVPN tunnel
ip address 172.17.5.1 255.255.255.0
no ip redirects
ip flow ingress
ip nhrp authentication *****
ip nhrp map multicast dynamic
ip nhrp network-id 1001
ip ospf network point-to-multipoint
ip ospf cost 100
ip ospf hello-interval 10
ip ospf mtu-ignore
load-interval 30
tunnel source x.x.x.x
tunnel mode gre multipoint
tunnel key 1101
end

On Spoke

interface Tunnel1101
description Multinet Link to HUB
ip address 172.17.5.44 255.255.255.0
ip flow ingress
ip nhrp authentication ****
ip nhrp map 172.17.5.1 x.x.x.x
ip nhrp map multicast x.x.x.x
ip nhrp network-id 1001
ip nhrp nhs 172.17.5.1
ip nhrp registration timeout 35
ip ospf network point-to-point
ip ospf cost 110
ip ospf mtu-ignore
tunnel source a.b.c.d
tunnel destination x.x.x.x
tunnel key 1101

One thing i want to re-emphasize, that everything is working fine, usually it happens when router reloads but it has also occured while working as well. Now as soon as i delete the tunnel and recreate it with different tunnel number, adjcaencies come up !!!

I was thinking it was some sort of bug !

[Peter Paluch]

Dear Jonn,

I have noticed an interesting issue in your configuration and I'd like to discuss it with you. On your spoke router, you seem to be using a point-to-point GRE tunnel. Is that a purpose? With point-to-point GRE tunnel on your spoke, you cannot have a spoke-to-spoke tunnel and all inter-spoke traffic must go through the hub router, thereby defeating one of major advantages of the DMVPN solution.

Allow me please a few more questions:

1. Which router is seen in the INIT state? A hub or a spoke?
2. Are the NHRP mappings and the CEF table absolutely correct both on the hub and the spoke in the moment of the problem?
3. Have you tried clearing the NHRP mappings or the entire CEF table and see whether the problem is removed?
4. Will you be able to peform the debug commands I've suggested in my previous post?

I admit, this does look like a bug but I have a feeling that something is triggering it and I would like to see where exactly does this bug appear and whether it can manifest itself also in some show commands, not just by OSPF adjacencies falling.

Best regards,

Peter

[wzhang]

Hi,

We have seen similar issues in the past due to IOS bugs, one of the more recent one is this http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsv43385. Note even though in the bug it describes removing and re-adding the same tunnel interface can be used as a workaround, it's not guaranteed given the problem is a caused by an internal timing condition in the code. This bug does affect 12.4(20)T and 12.4(22)T, although it should be fixed in 12.4(22)T2. I would suggest you upgrade the code to at least get pass this bug. If you still see the same issue, you should check to make sure the CEF adjacency is consistent with the NHRP mappings, and you can do this by "show adj tunnelx internal" to make the adj encap is the same as that in "show ip nhrp". However, the adjacency rewrite information is in hex, so you'd have to look for it and it's probably a better idea to open a TAC case to address it at that point.

Hope this helps,

Thanks,

Wen

[Peter Paluch]

Hello Wen,

Thank you very much for updating this thread. This looks like a very similar issue. It is always the insider's info like yours that makes the positive difference.

Best regards,

Peter

[Jonn cos]

Dear Wzhang and Peter,

I really want to thank you guys for helping me out. I will change the IOS today and will update you tomorrow if it solves the issue. Also Peter, just in case if the issue persists, i will paste all the debugs here and will take your described actions to see where the issue lies. Regarding Spoke end P2P GRE, yes its our requirement since we are using centralized servers, and spokes doesnt need to cummunicate with any other spoke.

Thanks alot again, i really appreciate it.