GRE tunnel flapping issue.

[sutharhemant90]

We are facing issue continues IPsec GRE tunnel flapping in case of backhaul link flapped. CPU going at the higher side and GRE Tunnels which are not enabled with crypto are stable.

1. We have DC DR Solution
2. Branch to DC GRE tunnel created for ipsec.
3. DC core router are connected with ISP on BGP.
4. For branch we have used static route for DC WAN ip pool
5. Between DC to DR Point point link running over eigrp
6. Redistribution between eigrp to bgp and bgp to eigrp on DC and DR.

[MHM Cisco World]

Check below 

MHM

[Peter Paluch]

Hello everyone,

Let's not jump to unwarranted conclusions, @MHM Cisco World - we don't even know if what is flapping are interfaces themselves, or just some protocol running over them; we do not know enough about the tunnels and their configuration; we do not know if they are using any GRE tunnel keys already... in fact, GRE tunnel keys cannot cause a tunnel to flap. At most, they can cause it to never come up, but flapping as a process of coming up and down can not be traced down to GRE keys or absence of them.

Hence, let's hold back on the compulsion to post a reply at every cost, and instead, let's collect actionable hard evidence beforehand.

@sutharhemant90 , I would like to ask you to share details from the device(s) where you see the tunnel flaps. I understand if some information cannot be shared openly - but please try to share as much as possible, anonymizing any sensitive information wherever needed.

1. What is the platform and the operating system version on the device?
2. Can you share the complete show logging from the device? Please note - I'm looking for all logging messages, even those that may appear unrelated.
3. Can you share the configuration of the device? The complete anonymized configuration would be ideal; at the very least, though, I would like to see the configuration of all tunnels, affected and unaffected, and the configuration of the BGP and EIGRP routing protocols including any route-maps, prefix lists or ACLs they may be referring to.

Thank you! Let's see if this information can move us further.

Best regards,
Peter

[sutharhemant90]

@Peter Paluch  I have pasted some basic configuration from core router (IPs are changed*). I will check for logs and update.

Tunnel with IPsec Crypto Enabled. (Flapping tunnel when any ISP link going up\down)

interface Tunnel1001
bandwidth 4096
ip address x.x.x.x 255.255.255.252
tunnel source x.x.x.x
tunnel mode ipip
tunnel destination x.x.x.x
tunnel protection ipsec profile GRE

Tunnel without IPsec Crypto Enabled. (Stable tunnel even if any isp links going up\down)

interface Tunnel1017
bandwidth 4096
ip address x.x.x.x 255.255.255.252
tunnel source x.x.x.x
tunnel mode ipip
tunnel destination x.x.x.x

-

router eigrp 110
variance 2
network 10.0.0.0
redistribute static
redistribute ospf 10 metric 512 1 255 1 1500
redistribute bgp 64940 metric 22528 1 255 1 1500

router bgp 64940
bgp log-neighbor-changes
neighbor 172.16.2.1 remote-as 9729
neighbor 172.16.2.1 description # BGP for ISP 1 #
neighbor 172.29.41.218 remote-as 55411
neighbor 172.29.41.218 description # BGP for ISP 1 #
!
address-family ipv4
redistribute static
redistribute eigrp 110
neighbor 172.16.2.1 activate
neighbor 172.16.2.1 weight 40000
neighbor 172.16.2.1 soft-reconfiguration inbound
neighbor 172.16.2.1 distribute-list ISP-1 in
neighbor 172.16.2.1 filter-list 70 out
neighbor 172.29.41.218 activate
neighbor 172.29.41.218 weight 40000
neighbor 172.29.41.218 soft-reconfiguration inbound
neighbor 172.29.41.218 distribute-list ISP-2 in
neighbor 172.29.41.218 filter-list 70 out
default-information originate
exit-address-family

Cisco Version 17.10.1a
Cisco - ISR4461/K9

[Peter Paluch]

Hello Hemant,

Thank you for the information shared so far.

From what I can tell so far, your configuration is not GRE. Rather, you are using IP-in-IP tunnels, optionally with IPsec protection. It is not a problem in itself, but I wanted to point that out so that we share the common understanding of the details of your configuration.

From the notes above, you wrote that the IPsec-protected tunnels flap whenever any ISP link goes up or down. This is important - is the ISP link going up or down the only trigger for that tunnel flap?

I'll still wait for the logs.

Best regards,
Peter

[MHM Cisco World]

Tunnel without IPSec profile dont use any kind of tunnel health check so that it always UP whenever destiantion is reachable (there is defualt route in rib)

Tunnel with IPsec profile use isakmp keepalive and when keepalive is failed the tunnel is down even if destination is reachable.

So your issue in your ISP not your device.

To be sure add ip sla to your defualt route and check the health of it.

MHM

[sutharhemant90]

There are rare incidents when a tunnel starts flapping automatically without any connected link flapping issue. If there is flapping or if a down link comes up at the data center during this time, my tunnel will start flapping, leading to high CPU utilization. To address this, I have created a script to bring all GRE tunnels down and then start a few tunnels step by step.

For testing purposes, I have applied the following CLI to one branch where the tunnel is stable. However, I am still unsure about the potential unknown impacts and whether it is recommended to use this approach

crypto isakmp keepalive 3600 60 periodic

Tunnel are not stable wherever keep alive timer is 20 10 Periodic

[MHM Cisco World]

you mention GRE in your original post
and I think you typo use (please @sutharhemant90 confirm) 
tunnel mode IPIP 
remove it please 
for mode I will suggest mode depend on case you have 
which of below you have ?

Regarding ISP did you use IP sla with default route or check link flapping? 

[Peter Paluch]

@MHM Cisco World ,

you mention GRE in your original post
and I think you typo use
tunnel mode IPIP
remove it please

NO! Absolutely NOT!

MHM, you cannot just blindly suggest that people change their existing working configurations! What is far more likely is that Hemant believes his tunnels are GRE-based but what has been configured and working in his network is in reality IP-in-IP style of tunnels.

Unless we have this clarified, no configuration shall be changed, and you, MHM, please stop and think before posting. This is already the second time I am asking you in this thread to avoid jumping into conclusions and stop replying compulsively without making sure you are, FIRST, factually correct, SECOND, that you are joining the discussion rather than hijacking it, THIRD, that you are cooperating instead of playing solo.

Peter

[Peter Paluch]

Hello Hemant,

A humble reminder for the logs - if it is possible to share them.

Thank you!

Best regards,
Peter