Losing EIGRP adjacencies w/ DMVPN

[qnguyen8]

I am trying a new code, 12.3.11T2 on our routers and every night when traffic is low, the spoke router with the new code on it would lose it EIGRP adjacencies. All the other spoke that are running 12.3.6a run fine. And the only way to get the neighbors back is to reload the router. Even reconfiguring the EIGRP process does not bring it back. Any insight would be appreciated.

[noc]

I have had this exact same problem. I am running a single DMVPN with two 2691 routers running 12.3(12). I opened a case with TAC because the spoke was loosing its routes after anywhere from 8 to 15 hours and a reload would only fix the problem. It turned out the input queue on the tunnel interface was incrementing and once it reached the max would drop all other traffic including eigrp routes. Here is the interface stats.

Tunnel0 is up, line protocol is up

Hardware is Tunnel

Internet address is 10.255.255.2/24

MTU 1514 bytes, BW 1000 Kbit, DLY 10000 usec,

reliability 255/255, txload 6/255, rxload 6/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 193.28.89.9 (FastEthernet0/0), destination UNKNOWN

Tunnel protocol/transport multi-GRE/IP, key 0x186A0, sequencing disabled

Checksumming of packets disabled, fast tunneling enabled

Tunnel protection via IPSec (profile "SDM_Profile1")

Last input 00:00:00, output 00:00:02, output hang never

Last clearing of "show interface" counters never

Input queue: 186/1024/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/0 (size/max)

5 minute input rate 27000 bits/sec, 20 packets/sec

5 minute output rate 27000 bits/sec, 18 packets/sec

7122069 packets input, 1729019848 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

6496026 packets output, 1314657726 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

KLDN2691#

TAC suggested I change the queue limit to 1024 with the command "hold-queue 1024 in" under the Tunnel interface config. This will cause the interface to take a much longer period based on traffic to max out. I am working on grabbing an engineering release from TAC now. Hope that helps.

-Jake Gibb

jgibb@krollworldwide.com

[qnguyen8]

Thanks for the info. I'll give that a shot. Please let me know when you get that release.

Quan Nguyen

qnguyen@cci.edu

[damarcus.richards]

I ran into the same problem on this code as well. I downgraded to 12.3.8T5 on the 831. Did you try bouncing the Tunnel interface on the hub side? Is the Hub having any CPU EIGRP issues? They 2 may not be related, but I was able to stablize EIGRP w/ DMVPN. Also, one thing to look at is to see if the router running the 12.3.11T2 code 'eigrp topology is showing'. If it has another interface as a better route than the Tunnel interface, that can cause the routing table to send eigrp updates out of the other interface for the Tunnel subnet & cause the RTO to be quite high.

[noc]

I was using 12.3.8T5 before and because I am also using NAT route maps with multiple links and CBAC I started seeing stack traces which lead to a known bug. The engineer suggested that I get a non-public release 12.3(11.6)T which has been stable on the 1712 so far. For the 2600's I am still awaiting the final image from engineering.

-Jake Gibb

[qnguyen8]

Which code are you running on your routers now? I'm still having problems with 12.3.12. Tunnel input queue still increments. NAT not working correctly. It's not expiring old translations and I have to manually clear the translation tables. Thanks

[qnguyen8]

Which code are you running on your routers now? I'm still having problems with 12.3.12. Tunnel input queue still increments. NAT not working correctly. It's not expiring old translations and I have to manually clear the translation tables. Thanks

[noc]

I am still running 12.3.12 but just yesterday I was given access to an interrim release c2691-advsecurityk9-mz.123-12.10. I have uploaded it to my 2691 and will reload with the new image later today. I hope this will solve the issue with the 2691 with the encryption module. What has me confused is my head end 2691 WITHOUT the encryption module does not have this problem. I have mentioned this to Cisco but have not received a response. Do you have an encryption card on your router??

-Jake

[qnguyen8]

I have a 7204VXR at the head running 12.3.6a. That works fine. One of my spokes is one of the new 2821. Code is very limited on these, only offering 12.3.8T and 12.3.11T versions. They both have VPN accelerator cards. The hub is running fine and my other spokes running older 12.3.6a code run fine also. It's just the 2821 with new code not working right.

[noc]

I just loaded 12.3(6c) on the 2691 router and will be watching it to see if it has the same issues as 12.3(12). So far the input queue is 0 and it appears to be fine. The image I got from Cisco crashes and I'm waiting for their response.

-Jake

[qnguyen8]

Did TAC give you a bug ID for this? Trying to work with TAC on this end. What's terrible is the 2800s only have 12.3.8T or 12.3.11T. They both have the same problem.

[noc]

Yep.CSCsa43492.I am having the image posted again because I think the MD5 sum did not match and that is what caused the router to crash when trying to load it.

-Jake

[noc]

PLEASE READ - Cry engine problem.

I've tested what I think may be a fix and help isolate the problem. I noticed that on the 2691 routers with the cry engine accelerator card the input queue problem was noticed whereas the routers with the cry engine card it was not. I disabled the cry engine with the command no cry engine accel and the input queue stopped incrementing!! This has been confirmed on two routers (1712 and 2691) both with accelerator cards installed. Can someone please try this using the 12.11(12) IOS revision and report back to this conversation?

-Jake

[qnguyen8]

TAC posted up 12.3.12(10) for me but, it worked like crap. All my routers have VPN AIM mods in them. You can also disable CEF and the queue will stop incrementing. I tried it last night and it hasn't incremented today. TAC says the 12.3.13 code due out in March is suppose to fix these bugs.

[gfullage]

Sorry to come in on this a little late, I'll try and help out as much as I can.

First off, CSCsa43492 was a definate major bug with very high priority, and should be resolved in 12.3(12.10) mainline code. This is an interim release that will have to be specially posted for you. It will then be integrated into the 12.3(13) release which according to the current schedule, is due out around mid-Feb.

If noc@krollworldwide.com upgrades to the interim version on his/her routers then the problem should go away, and you should be able to enable your encryption card. We don't want you to run without the card since this will put extra load on your CPU. You mention that it crashed when you laoded it but you think it wasn't downloaded correctly, can you confirm that for us please.

If qnguyen8 only has 28xx's, then we'll need to get that bug integated into the T train, which currently it doesn't seem to be. The bug for this same issue on the T train is CSCsa45724, so let me investigate with the developers what's going on and I'll get back to you here. Can you also expand further on "it worked like crap", what exactly was crap?