8200 router cellular interface shutting down

[KMNRuser]

We have one of our remote sites connecting back to us using a Cisco C8200L-1N-4T.

This router is in a remote location, and the only service we could find out there was cellular.

We have the Cellular interface connected; using "ip address negotiated".

We have 4 Tunnels configured on the box, and 3 of those tunnels pass traffic, but the 4th one, when it tries to pass traffic, will shut down the cellular interface for a period of a few seconds, which takes down the other 3 tunnels, and then once the cellular interface comes back up, connectivity is restored.

Has anyone ever witnessed this behavior before?  What could cause something within the configuration of the one tunnel to shut down the interface when a ping is sent across it?

Thanks for any input!

KMNRUser

[KMNRuser]

Rick,

The Cellular 0/1/0 is from a Pilot router that i used to get a proof of concept working, which was successful. 

I simply use the Cellular 0/2/0 on the production box since that is where we have the connection. 

I can certainly go ahead and try to use private internal for it and see if that works..ill follow up in a bit.

[Richard Burts]

Thanks for the clarification about the cellular interface(s). Interesting that the proof of concept was successful. Was it using Public IPs? Let us know the result of your trying private addressing.

[KMNRuser]

Rick,

Yes i did use the public IP's in the proof of concept.  It came up without issue.

I am struggling a bit trying to understand now how to shift to using the private IP's.. from looking at the Tunnel interfaces that are working, it seems my predecessor used some loopbacks and called them into the numbering so i am going to attempt to try that..

[MHM Cisco World]

more than 4 Seq of IPsec Map make the port down. 
use instead dynamic and hence you use one Seq BUT here the IPsec Peer can only initiate the traffic your router can not 

https://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/14131-ios-804.html

MHM

[Richard Burts]

I am surprised at the suggestion that more than 4 seq in the crypto map would create a problem. I have configured Cisco routers with crypto maps with much more than 4 sequences and not had a problem.

[MHM Cisco World]

Router#show platform cerm-information

maybe this command can help us to see his router limit and available tunnel  

MHM

[Dan Frey]

VZW can have source IP address violations and when that happens the cellular interface will bounce.   The cellular network expects all packets from the CPE to have the correct source addresses (from IP address negotiate) and if another IP address is the source the interface will bounce.   I see you have two cellular interfaces configured so it is possible that you are sourcing the tunnel from interface X and routing is telling it to egress interface Y which would cause the interface to bounce.   Not sure if this is your issue but something to investigate.  

[KMNRuser]

Dan,  

Thank you for your response!

The Cellular interface is currently configured and to the best of my understanding gets its IP from the provider by virtue of the statement configured on the interface, which is "ip address negotiated".  Our remote cellular routers all get the same address range from verizon, which is (sanitized) 10.XXX.XXX.<node IP for each is different>.  I used this on a Pilot router that we have as spare in that same range and everything is still working.  So your saying that if the provider sees a source address coming from us that is different than (10.XXX.XXX.<node>) it will drop the interface?

[Dan Frey]

Yes, that is correct.   Additional details in this doc.   https://www.cisco.com/c/dam/en/us/td/docs/routers/access/interfaces/software/deployment/guide/guide_c07-720271.pdf

[KMNRuser]

Dan,

Thanks for all of your input.  What i cant quite comprehend then is how my Pilot router is working without issue going back to the same host.  I have read the provided document, but i do not have any of that on the Pilot box, and i can ping off of the Pilot box regular or with an extended command sourcing it from a Gig (LAN) interface, and unlike the box i have in production which will drop the cellular interface each time for a short duration, the Pilot box never drops the cellular..

[Richard Burts]

KMNRUser

I wonder how significant it is that the previous tunnels were DMVPN and the new tunnel is ISAKMP. I wonder whether part of the issue might be the use of mode transport. If you change/remove that does the behavior change? Also if you change/correct the acl used does the behavior change?

When you attempt to bring up the new tunnel are there any log messages generated that might shed some light?

[KMNRuser]

Rick,

I wanted to get back to you and let you know that i did change the tunnel mode from transport to tunnel on both sides.  Unfortunately that did not resolve the issue.  One thing i am struggling with is the persistent logging.  I would think that i would have data in the logs that would indicate what fails when i apply the "match address GREINIPSEC" into the crypto map, but so far i dont see anything..it is almost as if the router isnt logging the data even though i have logging set at debugging level, and also have several debugs turned on including debug cry ipsec and debug cry isa..

[MHM Cisco World]

What Mr. @Dan Frey  mention and from your config 

I see issue' 

You use tunnel x using cellular 1 as tunnel source' and tunnel destination is y.y.y.y

If the RIB for y.y.y.y using cellular 2 not cellular 1 then the interface will drop.

To be sure the last tunnel you add' use it tunnel destination in 

Show ip route y.y.y.y longest 

See which interface is egress if it different than tunnel source then it can issue and as @Dan Frey mention the interface will down.

MHM

[KMNRuser]

That is my bad MHM Cisco World.  I think i confused everyone by the way i did the posting.  On the production box we are only using Cellular 0/2/0.  The Cellular 0/1/0 was from a Pilot router that we had as spare that i have working in a proof of concept configuration..I did not have access over the Holidays to the prod network, but i had a snippet on a notepad on my LT from the Pilot config and just pasted that in since the config matches other than the interface number.  I did not mean to send you down a rabbit hole!  Thanks for your response!

[KMNRuser]

I have ran some debugs on the 1100 router to try and determine why we loose connectivity when we apply the ACL "GREINIPSEC" to the crypto map which is assigned on interface Ce 0/2/0.  I can see where ISAKMP is failing, but am not crystal clear what the error 64 represents..(this is just a small snippet of the log).  I am curious if any of you have seen this before?

*Jan 3 14:42:34.309: ISAKMP-ERROR: (1003):IPSec policy invalidated proposal with error 64
*Jan 3 14:42:34.310: ISAKMP-ERROR: (1003):phase 2 SA policy not acceptable! (local 123.123.234.46 remote 10.140.0.20)
*Jan 3 14:42:34.310: ISAKMP: (1003):set new node 3380325336 to QM_IDLE
*Jan 3 14:42:34.310: ISAKMP: (1003):Sending NOTIFY PROPOSAL_NOT_CHOSEN protocol 3
spi 9223512672331109088, message ID = 3380325336
*Jan 3 14:42:34.310: ISAKMP-PAK: (1003):sending packet to 10.140.0.20 my_port 500 peer_port 500 (R) QM_IDLE
*Jan 3 14:42:34.310: ISAKMP: (1003):Sending an IKE IPv4 Packet.
*Jan 3 14:42:34.310: ISAKMP: (1003):purging node 3380325336
*Jan 3 14:42:34.310: ISAKMP-ERROR: (1003):deleting node 1559969754 error TRUE reason "QM rejected"
*Jan 3 14:42:34.311: ISAKMP: (1003):Node 1559969754, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
*Jan 3 14:42:34.311: ISAKMP: (1003):Old State = IKE_QM_READY New State = IKE_QM_READY
*Jan 3 14:42:44.821: ISAKMP: (1003):purging node 1897423342
*Jan 3 14:43:05.076: ISAKMP-PAK: (1003):received packet from 10.140.0.20 dport 500 sport 500 Global (R) QM_IDLE
*Jan 3 14:43:05.076: ISAKMP: (1003):set new node 2673259961 to QM_IDLE
*Jan 3 14:43:05.076: ISAKMP: (1003):processing HASH payload. message ID = 2673259961
*Jan 3 14:43:05.076: ISAKMP: (1003):processing SA payload. message ID = 2673259961
*Jan 3 14:43:05.076: ISAKMP: (1003):Checking IPSec proposal 1
*Jan 3 14:43:05.076: ISAKMP: (1003):transform 1, ESP_AES
*Jan 3 14:43:05.076: ISAKMP: (1003): attributes in transform:
*Jan 3 14:43:05.076: ISAKMP: (1003): encaps is 1 (Tunnel)
*Jan 3 14:43:05.076: ISAKMP: (1003): SA life type in seconds
*Jan 3 14:43:05.076: ISAKMP: (1003): SA life duration (basic) of 3600
*Jan 3 14:43:05.076: ISAKMP: (1003): SA life type in kilobytes

I have found a document that discusses it here https://community.cisco.com/t5/security-knowledge-base/l2l-vpn-troubleshooting-quot-ipsec-policy-invalidated-proposal/ta-p/3115635 and will try to use it to figure out the issue.