Re: Losing my eigrp route table every 24 hrs.

jomo frank · ‎02-16-2019

Hello Expert,

I have three routers connect hub using pppoe , I notice after 24 hr elapses I am losing my eigrp routes.(when I do show route the are no entries ) This issue is happening to only one of my spoke routers.

In order to re populate the eigrp routing table a I just issue a shutdown wait a few seconds and then no shutdown on the dialer interface.Once I do this the eigrp table is re-populate .

I did a show log and the error below was notice.

: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet ha
s invalid spi for destaddr=10.252.248.xx, prot=50, spi=0x1AD290A7(450007207), s
rcaddr=10.252.248.xx, input interface=Dialer

Could anyone provide some guidance.

Regards

balaji.bandi · ‎02-16-2019

Eigrp more of relay on underlay infrastructure, IP connectivity.

Can you post more information, is your tunnel up ? are you able to ping peer side IP address (when you see eigrp routes not available).

when you shut and no shut, that means dialer connect reset, and trunnel re-establish here..so the problem resolves, since you have underlay fixed, so Eigrp establish neighbour and get the routes.

check the logs you will see more information.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

jomo frank · ‎02-16-2019

Hello Balaji

When this occurs the tunnel shows up but there is no neighbor been establish.
The fix I am doing is a manual intervention , but this should not work like this the tunnels should be torn down and re built seamlessly.
I am trying to understand what is error means and if it plays any part in error I outline the first post.

Regards

This is the log file
---------------------------
Feb 16 14:21:56.398: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Acc
ess2, changed state to up
to administratively down
Feb 16 14:24:10.610: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthe
rnet0, changed state to down
Feb 16 14:24:27.430: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state t
o up
Feb 16 14:24:28.430: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthe
rnet0, changed state to up
Feb 16 14:24:43.134: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, ch
anged state to up
Feb 16 14:24:43.134: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, ch
anged state to up
Feb 16 14:24:43.134: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, ch
anged state to up
Feb 16 14:24:46.062: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.27.12 (
Tunnel2) is up: new adjacency
Feb 16 14:24:47.986: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.28.12 (
Tunnel3) is up: new adjacency
Feb 16 14:24:49.182: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.26.12 (
Tunnel1) is up: new adjacency
Feb 16 19:08:31.105: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, ch
anged state to down
Feb 16 19:08:31.109: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.27.12 (
Tunnel2) is down: interface down
Feb 16 19:08:42.677: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has
invalid spi for destaddr=10.252.248.143, prot=50, spi=0xC22C199E(3257670046), s
rcaddr=10.252.248.55, input interface=Dialer0
Feb 16 19:27:28.613: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.27.12 (
Tunnel2) is up: new adjacency
Feb 16 20:26:34.856: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.26.12 (
Tunnel1) is down: Interface PEER-TERMINATION received
Feb 16 20:26:36.012: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.26.12 (
Tunnel1) is up: new adjacency
Feb 16 21:02:03.444: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.27.12 (
Tunnel2) is down: holding time expired
Feb 16 21:02:04.284: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, ch
anged state to down
Feb 16 21:04:14.336: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, ch
anged state to up
Feb 16 21:04:17.600: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.27.12 (
Tunnel2) is up: new adjacency
Feb 16 21:21:00.256: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (
10.10.10.100)
Feb 16 21:26:03.012: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.27.12 (
Tunnel2) is down: Interface PEER-TERMINATION received
Feb 16 21:26:04.580: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.27.12 (

Regards

Georg Pauwen · ‎02-17-2019

Hello,

what topoogy do you have, DMVPN phase (what) ? Post the configs of the hub and the spoke. Do the clocks match on both ends (check NTP settings) ?

jomo frank · ‎02-23-2019

Hello Georg,

I was away for a some period

I have two other router spoke routers in the environment and both are working okay just this spoke is an issue.

I manage to do some debugging and I have forward same as an attachment.

Please vet I am unsure what happening to this router.

Regards

balaji.bandi · ‎02-17-2019

adding to other post, do you have connectivity to internet, i know you have mentioned Tunnel up,

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

jomo frank · ‎02-23-2019

Hello all,

I was await for a few day.

I must mention I have two outer spokes in this Hub and spoke setup and the other two spokes are working with any issues..

I did some debug and I have included an attachment with same.

Hub router

Central#sh ver
Cisco IOS Software, C890 Software (C890-UNIVERSALK9-M), Version 15.2(4)M6, RELEA
SE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Wed 19-Mar-14 22:06 by prod_rel_team

ROM: System Bootstrap, Version 15.1(2r)T1, RELEASE SOFTWARE (fc1)

Central uptime is 1 week, 3 days, 5 hours, 48 minutes
System returned to ROM by power-on
System restarted at 08:35:00 Caracas Wed Feb 13 2019
System image file is "flash:c890-universalk9-mz.152-4.M6.bin"
Last reload type: Normal Reload
Last reload reason: power-on

Spoke

------------------

Amazonia_Mall#sh ver
Cisco IOS Software, C800 Software (C800-UNIVERSALK9-M), Version 15.3(3)M5, RELEA
SE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 04-Feb-15 11:24 by prod_rel_team

ROM: System Bootstrap, Version 15.2(3r)XC, RELEASE SOFTWARE (fc1)

Amazonia_Mall uptime is 1 hour, 10 minutes
System returned to ROM by reload at 13:10:52 Caracas Sat Feb 23 2019
System image file is "flash:c800-universalk9-mz.SPA.153-3.M5.bin"
Last reload type: Normal Reload
Last reload reason: Reload Command

Regrads

Deepak Kumar · ‎02-23-2019

Hi,

I found some error messages which are showing packet drops or connection issues:

*Feb 23 17:02:37.426: ISAKMP:(2003):DPD incrementing error counter (5/5)
*Feb 23 17:02:37.426: ISAKMP:(2003):peer 10.252.248.131 not responding!
*Feb 23 17:02:37.426: ISAKMP:(2003):peer does not do paranoid keepalives.
*Feb 23 17:02:37.426: ISAKMP:(2003):deleting SA reason "End of ipsec tunnel" state (R) QM_IDLE       (peer 10.252.248.131)

(0): sending packet to 10.252.248.131 my_port 500 peer_port 500 (I) MM_NO_STATE
*Feb 23 17:02:48.502: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Feb 23 17:02:54.690: ISAKMP:(2003):purging node -1490228489
*Feb 23 17:02:58.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Feb 23 17:02:58.502: ISAKMP (0): incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
*Feb 23 17:02:58.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Feb 23 17:02:58.502: ISAKMP:(0): sending packet to 10.252.248.131 my_port 500 peer_port 500 (I) MM_NO_STATE
*Feb 23 17:02:58.502: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Feb 23 17:03:04.690: ISAKMP:(2003):purging node -1412752614
*Feb 23 17:03:08.502: ISAKMP: set new node 0 to QM_IDLE      
Feb 23 17:03:08.502: ISAKMP:(0):SA is still budding. Attached new ipsec request to it. (local 10.252.248.55, remote 10.252.248.131)
*Feb 23 17:03:08.502: ISAKMP: Error while processing SA request: Failed to initialize SA
*Feb 23 17:03:08.502: ISAKMP: Error while processing KMI message 0, error 2.

Feb 23 17:02:38.502: ISAKMP:(0):found peer pre-shared key matching 10.252.248.131
*Feb 23 17:02:38.502: ISAKMP:(0): constructed NAT-T vendor-rfc3947 ID
*Feb 23 17:02:38.502: ISAKMP:(0): constructed NAT-T vendor-07 ID
*Feb 23 17:02:38.502: ISAKMP:(0): constructed NAT-T vendor-03 ID
*Feb 23 17:02:38.502: ISAKMP:(0): constructed NAT-T vendor-02 ID
*Feb 23 17:02:38.502: ISAKMP:(0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
*Feb 23 17:02:38.502: ISAKMP:(0):Old State = IKE_READY  New State = IKE_I_MM1 
*Feb 23 17:02:38.502: ISAKMP:(0): beginning Main Mode exchange
*Feb 23 17:02:38.502: ISAKMP:(0): sending packet to 10.252.248.131 my_port 500 peer_port 500 (I) MM_NO_STATE
*Feb 23 17:02:38.502: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Feb 23 17:02:45.014: ISAKMP:(2003):purging node -1543044096
*Feb 23 13:02:45: %DIALER-6-BIND: Interface Vi2 bound to profile Di0
*Feb 23 13:02:45: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to up
*Feb 23 13:02:45: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to up
*Feb 23 17:02:48.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Feb 23 17:02:48.502: ISAKMP (0): incrementing error counter on sa, attempt 1 of 5: retransmit phase 1
*Feb 23 17:02:48.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Feb 23 17:02:48.502: ISAKMP:(0): sending packet to 10.252.248.131 my_port 500 peer_port 500 (I) MM_NO_STATE
*Feb 23 17:02:48.502: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Feb 23 17:02:54.690: ISAKMP:(2003):purging node -1490228489
*Feb 23 17:02:58.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Feb 23 17:02:58.502: ISAKMP (0): incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
*Feb 23 17:02:58.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Feb 23 17:02:58.502: ISAKMP:(0): sending packet to 10.252.248.131 my_port 500 peer_port 500 (I) MM_NO_STATE
*Feb 23 17:02:58.502: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Feb 23 17:03:04.690: ISAKMP:(2003):purging node -1412752614
*Feb 23 17:03:08.502: ISAKMP: set new node 0 to QM_IDLE      
Feb 23 17:03:08.502: ISAKMP:(0):SA is still budding. Attached new ipsec request to it. (local 10.252.252.55, remote 10.252.248.131)
*Feb 23 17:03:08.502: ISAKMP: Error while processing SA request: Failed to initialize SA
*Feb 23 17:03:08.502: ISAKMP: Error while processing KMI message 0, error 2.
*Feb 23 17:03:08.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Feb 23 17:03:08.502: ISAKMP (0): incrementing error counter on sa, attempt 3 of 5: retransmit phase 1
*Feb 23 17:03:08.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Feb 23 17:03:08.502: ISAKMP:(0): sending packet to 10.252.248.131 my_port 500 peer_port 500 (I) MM_NO_STATE
*Feb 23 17:03:08.502: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Feb 23 17:03:18.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Feb 23 17:03:18.502: ISAKMP (0): incrementing error counter on sa, attempt 4 of 5: retransmit phase 1
*Feb 23 17:03:18.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Feb 23 17:03:18.502: ISAKMP:(0): sending packet to 10.252.248.131 my_port 500 peer_port 500 (I) MM_NO_STATE
*Feb 23 17:03:18.502: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Feb 23 17:03:28.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Feb 23 17:03:28.502: ISAKMP (0): incrementing error counter on sa, attempt 5 of 5: retransmit phase 1
*Feb 23 17:03:28.502: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Feb 23 17:03:28.502: ISAKMP:(0): sending packet to 10.252.248.131 my_port 500 peer_port 500 (I) MM_NO_STATE
*Feb 23 17:03:28.502: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Feb 23 17:03:37.426: ISAKMP:(2003):purging SA., sa=EF0F2AC, delme=EF0F2AC
*Feb 23 17:03:38.514: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Feb 23 17:03:38.514: ISAKMP:(0):peer does not do paranoid keepalives.
*Feb 23 17:03:38.514: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer 10.252.248.131)
*Feb 23 17:03:38.514: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer 10.252.248.131) 
*Feb 23 17:03:38.514: ISAKMP: Unlocking peer struct 0xEECFF50 for isadb_mark_sa_deleted(), count 0
*Feb 23 17:03:38.514: ISAKMP: Deleting peer node by peer_reap for 10.252.248.131: EECFF50

Meanwhile, it is good if you will share logs from the remote site also.

Check your NAT configuration and you must allow the port 500 and 4500 for successful communication. If you are sure that there is no issue with your device then check the site to site communication with the help of IPSLA.

Regards,

Deepak Kumar

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

jomo frank · ‎02-24-2019

Hello Deepak,

I am not using nat since there is no connection to the internet, just a gre tunnel to remote site.

I not sure I can run the debug for the remote site which is hub(production) which is connected to 6 other spoke---l this may overload the router?

When you say connection issue do you mean the Service provider link from this site to my hub is unstable, because since the issue started the Service provider has commence monitoring the link and to-date they report no time-out .

Regards

Deepak Kumar · ‎02-24-2019

Hi,

I not sure I can run the debug for the remote site which is hub(production) which is connected to 6 other spoke---l this may overload the router?

Yes, it is but you can take conditional debug specific for the concern spoke. It will not put much load but check your CPU and RAM uses before start the conditional debugging on the hub.

As I said that Configure an IPSLA on spoke and monitor the connectivity between HUB and Spoke.

Edited:

I noticed some more logs as

Feb 16 14:24:10.610: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthe
rnet0, changed state to down
Feb 16 14:24:27.430: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state t
o up
Feb 16 14:24:28.430: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthe
rnet0, changed state to up

Means your WAN connection went down. Check your connection as well.

Regards,

Deepak kumar

Regards,

Deepak Kumar

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

jomo frank · ‎02-24-2019

Hello Deepak,

The drop you notice are fine during the course of day they may be one or two brief drops but the tunnel recovered

see below is an instance when this occurs

*Feb 23 16:16:36: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.19.10 (Tunnel2) is down: holding time expired
*Feb 23 16:16:36: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down
*Feb 23 16:17:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to up
*Feb 23 16:17:03: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.19.10 (Tunnel2) is up: new adjacency
*Feb 24 07:10:43: %SYS-5-CONFIG_I: Configured from console by admin on vty1 (172.26.10.50)

The is what make this issue more weird the vpn goes down and recovers

But we notice after 24 hrs elapse after a manual intervention the vpn stays down until another manual intervention

The manual intervention is either shutdown and restarting a dialer interface or reload the router once this done the vpn is re-establish and works okay until the next 24 hrs then it goes down and refuse to re-establish until another manual intervention is done.

The last manual reload was done 23 Feb 2019 at time 13.16 as per debug logs I sent to you .

Just a few minutes ago at 13:16 the vpn tunnel went down and did not recover I had to manual reload the router the vpn tunnel was re-establish . See logs attached

Regards

balaji.bandi · ‎02-24-2019

Just reading all the thread messages now.

So what were the state of the device before reload, do we see any obnormal logs ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Georg Pauwen · ‎02-24-2019

Hello,

not sure if this has already been asked or posted, but have we seen the full configs of the hub and the problem spoke yet ? Are all spokes running the same IOS ?

jomo frank · ‎02-24-2019

Hello Georg,

I included an attachment with config of hub and spoke. The spokes Kitty and South Ruimveldt are okay.

Regards

Georg Pauwen · ‎02-24-2019

Hello,

looking at your configs, my suggestion of using static EIGRP neighbors is not a good idea, since you have multiple tunnels using the same outgoing interface...

The isakmp keepalive values on the hub and spoke do not match, on the spoke, make sure you use the same values as on the hub:

Amazonia_Mall

crypto isakmp keepalive 10 3 periodic