8200 router cellular interface shutting down

[KMNRuser]

We have one of our remote sites connecting back to us using a Cisco C8200L-1N-4T.

This router is in a remote location, and the only service we could find out there was cellular.

We have the Cellular interface connected; using "ip address negotiated".

We have 4 Tunnels configured on the box, and 3 of those tunnels pass traffic, but the 4th one, when it tries to pass traffic, will shut down the cellular interface for a period of a few seconds, which takes down the other 3 tunnels, and then once the cellular interface comes back up, connectivity is restored.

Has anyone ever witnessed this behavior before?  What could cause something within the configuration of the one tunnel to shut down the interface when a ping is sent across it?

Thanks for any input!

KMNRUser

[Richard Burts]

Thanks for the additional information. It does help clarify one thing, though I think it is not the most important part of the issue. In a previous post you showed the acl

ip access-list extended GREINIPSEC
10 permit gre any any

So the acl is looking for GRE packets. But the output shows that "my_port 500 peer_port 500" so the acl needs to look for isakmp rather than gre.

[Georg Pauwen]

Hello,

I have not followed the entire thread, so I might have missed that, but can you post the full running configs of BOTH endpoints ?

[KMNRuser]

Hi Georg,

Certainly!  I am attaching the sanitized config files for your reading pleasure..

Thank  you,

KMNRUser

[Dan Frey]

Your debug in a previous message looks like IKE is coming up (QM_IDLE) and IPSEC phase2 is failing.   Are the transform set aligned (tunnel mode or transport mode) on each of the endpoints?   Is one tunnel and the other transport?

Under "interface tunnel XX" the tunnel mode needs to align on each side.   Default is GRE if nothing is specified and can be changed to IPSEC with "tunnel mode ipsec ipv4".   Transport mode in the transform set should be used for GRE tunnel;  Tunnel mode in the transform set should be used with IPSEC.   Each end needs to have the settings the same.   

[KMNRuser]

Dan,  

I have the tunnel mode set as transport on each side.  Another Community member had recently suggested that perhaps moving the mode to tunnel from transport may help.  We are in a change freeze today but tomorrow i plan on switching the mode and responding back with the results.

[Georg Pauwen]

Hello,

which tunnels in these configs are the ones failing ? Or is it arbitrary (three work, any fourth one fails) ?

My first thought was that this could be a traffic volume issue, but looking at your configs, I am not sure it even applies. Can you configure the below command on both routers (use the maximum matching vaue):

crypto ipsec security-association lifetime kilobytes

[KMNRuser]

Hello Georg,

Interface Tunnel 6 is the tunnel i am trying to configure and pass traffic thru.

The maximum value for the ipsec security association lifetime in kb shows as follows:

SideA_RTR(config)#cry ipsec security-association ?
dummy Enable transmitting dummy packets
ecn Handling of ECN bit
idle-time Automatically delete IPSec SAs after a given idle period.
lifetime security association lifetime
multi-sn Enable multiple sequence number per IPSec SA
replay Set replay checking.

SideA_RTR(config)#cry ipsec security-association lifetime ?
days Time-based key duration in days
kilobytes Volume-based key duration
seconds Time-based key duration in seconds

SideA_RTR(config)#cry ipsec security-association lifetime kilobytes ?
<2560-4294967295> Security association duration in kilobytes encrypted
disable Disable Volume-based Rekey

SideA_RTR(config)#cry ipsec security-association lifetime kilobytes

I want to make sure that when implementing this command, that i do not cause further harm.

What risks am I taking when i turn this specific command up to its max value, which appears to be 4294967295?

Thank you!

ODEC_COMM_RTR(config)#cry ipsec security-association ?
dummy Enable transmitting dummy packets
ecn Handling of ECN bit
idle-time Automatically delete IPSec SAs after a given idle period.
lifetime security association lifetime
multi-sn Enable multiple sequence number per IPSec SA
replay Set replay checking.

ODEC_COMM_RTR(config)#cry ipsec security-association lifetime ?
days Time-based key duration in days
kilobytes Volume-based key duration
seconds Time-based key duration in seconds

ODEC_COMM_RTR(config)#cry ipsec security-association lifetime

ODEC_COMM_RTR(config)#cry ipsec security-association ?
dummy Enable transmitting dummy packets
ecn Handling of ECN bit
idle-time Automatically delete IPSec SAs after a given idle period.
lifetime security association lifetime
multi-sn Enable multiple sequence number per IPSec SA
replay Set replay checking.

ODEC_COMM_RTR(config)#cry ipsec security-association lifetime ?
days Time-based key duration in days
kilobytes Volume-based key duration
seconds Time-based key duration in seconds

ODEC_COMM_RTR(config)#cry ipsec security-association lifetime kilobytes ?
<2560-4294967295> Security association duration in kilobytes encrypted
disable Disable Volume-based Rekey

ODEC_COMM_RTR(config)#cry ipsec security-association lifetime kilobytes

kilobytes ?
<2560-4294967295> Security association duration in kilobytes encrypted
disable Disable Volume-based Rekey

ODEC_COMM_RTR(config)#cry ipsec security-association lifetime kilobytes 4294967295

[Georg Pauwen]

Hello,

I have pieced together the parts of the NAME_CHAD_RTR router that tunnel 6 uses. The access list that is matched in the crypto map exists, but is empty. Can you check if that might be the problem ?

interface Tunnel6
ip address 10.101.6.2 255.255.255.0
tunnel source Cellular0/2/0
tunnel destination XXXXX7.248.46
!
interface Cellular0/2/0
description ***** VzW EVDO Interface *****
ip address negotiated
ip tcp adjust-mss 1388
dialer in-band
dialer idle-timeout 60
dialer watch-group 1
dialer-group 1
ipv6 enable
pulse-time 1
crypto map ICCP_BACKUP
ip virtual-reassembly
!
crypto map ICCP_BACKUP 81 ipsec-isakmp
! Incomplete
set peer XXXXX7.248.46
set transform-set CHADunityTS
match address GREINIPSEC
!
crypto ipsec transform-set CHADunityTS esp-aes 256 esp-sha512-hmac
mode tunnel
!
--> ip access-list extended GREINIPSEC

[KMNRuser]

Georg,

It is actually not empty, but rather has the value "permit gre any any" as its only ACE..

[Georg Pauwen]

Hello,

are you sure ? The crypto map says 'Incomplete', that is usually because something is missing. I pasted the config you posted (maybe you deleted the entry when you sanitized the config ?)...

Building configuration...
Current configuration : 24931 bytes
!
! Last configuration change at 09:55:37 EST Mon Jan 8 2024 by sp3ar-m1nt19
!
version 17.6
service tcp-keepalives-in
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
service call-home
platform qfp utilization monitor load 80
platform punt-keepalive disable-kernel-core
!
hostname NAME_CHAD_RTR
!
boot-start-marker
boot system flash bootflash:c8000be-universalk9.17.06.01a.SPA.bin
boot-end-marker
!
!
logging buffered 20000000
logging persistent url bootflash:/LOGS size 104857600 filesize 5242880
enable secret 9 $14$xqrD$yBMUDszwNogsVE$1Vy8p4gfx5wt/xyB0DSef3t9vi7vSJ6MRy.X9QaIsjo
!
aaa new-model
!
!
aaa authentication login default group tacacs+ local
aaa authentication login telnet group tacacs+ local enable
aaa authentication login EEMScript none
aaa authentication enable default enable
aaa authorization config-CHADands
aaa authorization exec EEMScript none
aaa authorization CHADands 0 default group tacacs+ none
aaa authorization CHADands 0 EEMScript none
aaa authorization CHADands 1 default group tacacs+ none
aaa authorization CHADands 1 EEMScript none
aaa authorization CHADands 2 default group tacacs+ none
aaa authorization CHADands 3 default group tacacs+ none
aaa authorization CHADands 4 default group tacacs+ none
aaa authorization CHADands 5 default group tacacs+ none
aaa authorization CHADands 6 default group tacacs+ none
aaa authorization CHADands 7 default group tacacs+ none
aaa authorization CHADands 8 default group tacacs+ none
aaa authorization CHADands 9 default group tacacs+ none
aaa authorization CHADands 10 default group tacacs+ none
aaa authorization CHADands 11 default group tacacs+ none
aaa authorization CHADands 12 default group tacacs+ none
aaa authorization CHADands 13 default group tacacs+ none
aaa authorization CHADands 14 default group tacacs+ none
aaa authorization CHADands 15 default group tacacs+ none
aaa authorization CHADands 15 telnet group tacacs+ local
aaa authorization CHADands 15 EEMScript none
aaa authorization network default group tacacs+ none
aaa accounting CHADands 0 default start-stop group tacacs+
aaa accounting CHADands 1 default start-stop group tacacs+
aaa accounting CHADands 2 default start-stop group tacacs+
aaa accounting CHADands 3 default start-stop group tacacs+
aaa accounting CHADands 4 default start-stop group tacacs+
aaa accounting CHADands 5 default start-stop group tacacs+
aaa accounting CHADands 6 default start-stop group tacacs+
aaa accounting CHADands 7 default start-stop group tacacs+
aaa accounting CHADands 8 default start-stop group tacacs+
aaa accounting CHADands 9 default start-stop group tacacs+
aaa accounting CHADands 10 default start-stop group tacacs+
aaa accounting CHADands 11 default start-stop group tacacs+
aaa accounting CHADands 12 default start-stop group tacacs+
aaa accounting CHADands 13 default start-stop group tacacs+
aaa accounting CHADands 14 default start-stop group tacacs+
aaa accounting CHADands 15 default start-stop group tacacs+
aaa accounting network default start-stop group tacacs+
aaa accounting connection default start-stop group tacacs+
aaa accounting system default start-stop group tacacs+
!
!
!
!
!
!
aaa session-id CHADon
clock timezone EST -5 0
clock summer-time EDT recurring
!
!
!
!
!
!
!
ip name-server 8.8.8.8
ip domain name NAME.fire.domain
!
!
!
login on-success log
!
!
!
!
!
!
!
subscriber templating
!
!
!
!
!
!
multilink bundle-name authenticated
!
flow record NAME_CHAD_RTRrecord
match ipv4 source address
match ipv4 destination address
match ipv4 protocol
match transport source-port
match transport destination-port
match ipv4 tos
match interface input
collect interface output
collect counter bytes
collect counter packets
collect timestamp sys-uptime first
collect timestamp sys-uptime last
collect application name
collect routing source as
collect routing destination as
!
!
flow exporter NAME_CHAD_RTRexport
destination 192.168.1.146
source GigabitEthernet0/0/0
transport udp 2055
template data timeout 60
option application-table timeout 60
option application-attributes timeout 300
!
!
flow monitor NAME_CHAD_RTRMonitor
exporter NAME_CHAD_RTRexport
cache timeout active 60
record NAME_CHAD_RTRrecord
!
!
!
!
!
!
!
!
!
!
crypto pki trustpoint TP-self-signed-3320455412
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3320455412
revocation-check none
rsakeypair TP-self-signed-3320455412
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
!
crypto pki certificate chain TP-self-signed-3320455412
certificate self-signed 01
30820330 30820218 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 33333230 34353534 3132301E 170D3233 30393139 31323031
32365A17 0D333330 39313831 32303132 365A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 33323034
35353431 32308201 22300D06 092A8648 86F70D01 01010500 0382010F 00308201
0A028201 0100BA48 19971930 6B6D8BA5 6C2FEE64 38E9DA20 D0ABDE70 AC5F7466
C352EA11 35886A55 0EA22F46 FC01A878 3772EC35 2C1DF88D 37EE06F3 4204A1D5
F144818A 1634B384 27C8951B 2A2D6272 9980C919 79086294 BFDC8FEF D924EEBE
B5A52EC2 5C81B54F 3448EC4B 27C1EBAB 3D2F9F54 5870E84C 98D126FC 4CF4A3E0
316DE8B4 75BC9316 698D43CA DE252E39 A9ED1556 F8119A8A A29F378F E3470A9E
7006B4A8 004C39DE E8CA08B0 5D1791C7 D4A2799B 6A175D9E 4EBCEB4D E38F9F95
2BE9D160 911130D8 43ABD727 077417EB 48C75AF5 6BF85BFC A437B964 781C4CED
14C7A4B3 E08A417E 812D5077 0B94CA57 2EF4B3F2 003CFFBE AC79F378 CE72DA1A
D3BD467C C1B30203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF
301F0603 551D2304 18301680 145F5B78 A845CD3E 58AFB58D CA06FFD0 E7458986
57301D06 03551D0E 04160414 5F5B78A8 45CD3E58 AFB58DCA 06FFD0E7 45898657
300D0609 2A864886 F70D0101 05050003 82010100 00EBE200 CB8AA107 04C5F489
E43AB6DB D0D3E5A7 7ED9321F 3732D091 EE6A393F CA0A6A29 B1BD6D7A 870706E6
CC6FBB8A F55DEB46 1799EE04 506D3CF1 7FC5FD7D BF41FDEB 14F227BC 0967657D
D5CD4751 6AEC36AC 74E12B5C 1CE629B1 4469F0B2 D3C6B81A 37C67D5E 13177EEB
4254C309 066F077E D13115C1 3445A824 BFC33F8F 3213E53D D2A05DC2 2DB61A08
5BD91E54 F4F0A368 05B519BF 586B974E DC7870CD 110FE40B DC91D501 0680F8C9
D1679EB2 2D1B9209 4EB73784 3DD31712 9655A879 ABD5E7B4 97D2C8DA 27B8E950
0D3674D5 4142454C 22C85D75 0EAA31E6 E5DE0A85 6F38134C 08187436 5BC6382E
2AA0EBBD EB88FAF1 220F037D B0FD3AA4 2C4014DE
quit
crypto pki certificate chain SLA-TrustPoint
certificate ca 01
30820321 30820209 A0030201 02020101 300D0609 2A864886 F70D0101 0B050030
32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363
6F204C69 63656E73 696E6720 526F6F74 20434130 1E170D31 33303533 30313934
3834375A 170D3338 30353330 31393438 34375A30 32310E30 0C060355 040A1305
43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720
526F6F74 20434130 82012230 0D06092A 864886F7 0D010101 05000382 010F0030
82010A02 82010100 A6BCBD96 131E05F7 145EA72C 2CD686E6 17222EA1 F1EFF64D
CBB4C798 212AA147 C655D8D7 9471380D 8711441E 1AAF071A 9CAE6388 8A38E520
1C394D78 462EF239 C659F715 B98C0A59 5BBB5CBD 0CFEBEA3 700A8BF7 D8F256EE
4AA4E80D DB6FD1C9 60B1FD18 FFC69C96 6FA68957 A2617DE7 104FDC5F EA2956AC
7390A3EB 2B5436AD C847A2C5 DAB553EB 69A9A535 58E9F3E3 C0BD23CF 58BD7188
68E69491 20F320E7 948E71D7 AE3BCC84 F10684C7 4BC8E00F 539BA42B 42C68BB7
C7479096 B4CB2D62 EA2F505D C7B062A4 6811D95B E8250FC4 5D5D5FB8 8F27D191
C55F0D76 61F9A4CD 3D992327 A8BB03BD 4E6D7069 7CBADF8B DF5F4368 95135E44
DFC7C6CF 04DD7FD1 02030100 01A34230 40300E06 03551D0F 0101FF04 04030201
06300F06 03551D13 0101FF04 05300301 01FF301D 0603551D 0E041604 1449DC85
4B3D31E5 1B3E6A17 606AF333 3D3B4C73 E8300D06 092A8648 86F70D01 010B0500
03820101 00507F24 D3932A66 86025D9F E838AE5C 6D4DF6B0 49631C78 240DA905
604EDCDE FF4FED2B 77FC460E CD636FDB DD44681E 3A5673AB 9093D3B1 6C9E3D8B
D98987BF E40CBD9E 1AECA0C2 2189BB5C 8FA85686 CD98B646 5575B146 8DFC66A8
467A3DF4 4D565700 6ADF0F0D CF835015 3C04FF7C 21E878AC 11BA9CD2 55A9232C
7CA7B7E6 C1AF74F6 152E99B7 B1FCF9BB E973DE7F 5BDDEB86 C71E3B49 1765308B
5FB0DA06 B92AFE7F 494E8A9E 07B85737 F3A58BE1 1A48A229 C37C1E69 39F08678
80DDCD16 D6BACECA EEBC7CF9 8428787B 35202CDC 60E4616A B623CDBD 230E3AFB
418616A9 4093E049 4D10AB75 27E86F73 932E35B5 8862FDAE 0275156F 719BB2F0
D697DF7F 28
quit
!
!
!
!
!
!
!
!
!
license udi pid C8200L-1N-4T sn FJC27221BDT
license boot level network-essentials
memory free low-watermark processor 67522
!
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
username sp3ar-m1nt19 privilege 15 password 7 023C08492E16350342191C4A0911381F5B52262F14
!
redundancy
mode none
!
!
crypto ikev2 proposal NEW_AADSC
encryption aes-cbc-256
integrity sha384
group 20
crypto ikev2 proposal AADSC
encryption aes-cbc-256
integrity sha1
group 20
!
crypto ikev2 policy NEW_AADSC
proposal NEW_AADSC
crypto ikev2 policy AADSC
proposal AADSC
!
crypto ikev2 keyring AADSC
peer AADSC_PRI
address 10.2.4.100
pre-shared-key local fupasswd
pre-shared-key remote fupasswd
!
peer AADSC_SEC
address 192.168.15.200
pre-shared-key local gupasswd
pre-shared-key remote gupasswd
!
!
!
crypto ikev2 profile AADSC_PRI
match identity remote address 10.2.4.100 255.255.255.255
identity local fqdn NAME-CHAD-RTR.NAME.fire.domain
authentication remote pre-share
authentication local pre-share
keyring local AADSC
dpd 10 2 on-demand
!
crypto ikev2 profile AADSC_SEC
match identity remote address 192.168.15.200 255.255.255.255
identity local fqdn NAME-CHAD-RTR.NAME.fire.domain
authentication remote pre-share
authentication local pre-share
keyring local AADSC
dpd 10 2 on-demand
!
crypto ikev2 profile AADSC_BCC
match identity remote address 172.16.50.200 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local AADSC
dpd 10 2 on-demand
!
!
controller Cellular 0/2/0
!
!
!
!
class-map match-all AADSC
match access-group name AADSC
class-map match-all NAME-FTP
match access-group name NAME-FTP
match protocol ftp
!
policy-map NAME-CBWFQ
class AADSC
bandwidth percent 65
class NAME-FTP
bandwidth percent 10
class class-default
fair-queue
!
!
!
!
!
!
!
!
crypto isakmp policy 10
encryption aes
hash sha256
authentication pre-share
group 14
!
crypto isakmp policy 50
encryption 3des
hash md5
authentication pre-share
group 2
!
crypto isakmp policy 70
encryption 3des
hash md5
authentication pre-share
group 2
!
crypto isakmp policy 80
encryption 3des
hash md5
authentication pre-share
group 2
!
crypto isakmp policy 90
encryption 3des
hash md5
authentication pre-share
group 2
crypto isakmp key 6 rupassword address XXXXX7.248.46
crypto isakmp key zupasswd address 0.0.0.0
crypto isakmp keepalive 10
!
!
crypto ipsec transform-set NAME_WAN esp-3des esp-md5-hmac
mode tunnel
crypto ipsec transform-set AADSC esp-aes 256 esp-sha-hmac
mode tunnel
crypto ipsec transform-set CHADunityTS esp-aes 256 esp-sha512-hmac
mode tunnel
!
crypto ipsec profile AADSC_BCC
set transform-set AADSC
set ikev2-profile AADSC_BCC
!
crypto ipsec profile AADSC_PRI
set transform-set AADSC
set ikev2-profile AADSC_PRI
!
crypto ipsec profile AADSC_SEC
set transform-set AADSC
set ikev2-profile AADSC_SEC
!
!
!
crypto map ICCP_BACKUP 50 ipsec-isakmp
set peer XXXXX2.188.206
set transform-set NAME_WAN
match address 100
crypto map ICCP_BACKUP 70 ipsec-isakmp
set peer XXXXX4.249.218
set transform-set NAME_WAN
match address 100
crypto map ICCP_BACKUP 80 ipsec-isakmp
set peer XXXXX6.127.14
set transform-set NAME_WAN
match address 110
crypto map ICCP_BACKUP 81 ipsec-isakmp
! Incomplete
set peer XXXXX7.248.46
set transform-set CHADunityTS
match address GREINIPSEC
crypto map ICCP_BACKUP 90 ipsec-isakmp
set peer XXXXX5.254.26
set transform-set NAME_WAN
match address 120
!
!
!
!
!
!
!
!
interface Loopback0
ip address 192.168.150.4 255.255.255.255
!
interface Loopback100
ip address 10.101.1.8 255.255.255.255
!
interface Tunnel1
ip address 10.101.2.8 255.255.255.0
ip mtu 1400
ip nhrp authentication cisco
ip nhrp map 10.101.2.1 10.101.1.1
ip nhrp network-id 100000
ip nhrp holdtime 360
ip nhrp nhs 10.101.2.1
ip tcp adjust-mss 1360
delay 1000
tunnel source Cellular0/2/0
tunnel destination 10.101.1.1
tunnel key 100000
!
interface Tunnel2
ip address 10.101.3.8 255.255.255.0
ip mtu 1400
ip nhrp authentication cisco
ip nhrp map 10.101.3.1 10.101.1.254
ip nhrp network-id 100000
ip nhrp holdtime 360
ip nhrp nhs 10.101.3.1
ip tcp adjust-mss 1360
delay 1000
tunnel source Cellular0/2/0
tunnel destination 10.101.1.254
tunnel key 100000
!
interface Tunnel3
ip address 10.101.4.8 255.255.255.0
ip mtu 1400
ip nhrp authentication cisco
ip nhrp map 10.101.4.1 10.101.1.253
ip nhrp network-id 100000
ip nhrp holdtime 360
ip nhrp nhs 10.101.4.1
ip tcp adjust-mss 1360
delay 1000
tunnel source Cellular0/2/0
tunnel destination 10.101.1.253
tunnel key 100000
!
interface Tunnel4
ip address 10.101.5.8 255.255.255.0
!
interface Tunnel6
ip address 10.101.6.2 255.255.255.0
tunnel source Cellular0/2/0
tunnel destination XXXXX7.248.46
!
interface Tunnel11
ip address 10.12.200.14 255.255.255.252
ip nat inside
tunnel source Loopback0
tunnel mode ipsec ipv4
tunnel destination 10.2.4.100
tunnel protection ipsec profile AADSC_PRI
ip virtual-reassembly
!
interface Tunnel22
ip address 10.11.200.14 255.255.255.252
ip nat inside
shutdown
tunnel source Loopback0
tunnel mode ipsec ipv4
tunnel destination 192.168.15.200
tunnel protection ipsec profile AADSC_SEC
ip virtual-reassembly
!
interface Tunnel33
ip address 10.13.200.14 255.255.255.252
ip nat inside
tunnel source Loopback0
tunnel destination 192.168.150.60
ip virtual-reassembly
!
interface GigabitEthernet0/0/0
ip flow monitor NAME_CHAD_RTRMonitor input
ip flow monitor NAME_CHAD_RTRMonitor output
ip address 194.49.37.99 255.255.255.0
ip nat outside
negotiation auto
ip virtual-reassembly
!
interface GigabitEthernet0/0/1
description VZ_C
no ip address
shutdown
negotiation auto
!
interface GigabitEthernet0/0/2
no ip address
shutdown
negotiation auto
!
interface GigabitEthernet0/0/3
no ip address
shutdown
negotiation auto
!
interface Cellular0/2/0
description ***** VzW EVDO Interface *****
ip address negotiated
ip tcp adjust-mss 1388
dialer in-band
dialer idle-timeout 60
dialer watch-group 1
dialer-group 1
ipv6 enable
pulse-time 1
crypto map ICCP_BACKUP
ip virtual-reassembly
!
interface Cellular0/2/1
no ip address
shutdown
!
!
router eigrp 13
distribute-list Member-Routes out Tunnel1
distribute-list Member-Routes out Tunnel3
distribute-list Member-Routes out Tunnel2
network 10.101.2.0 0.0.0.255
network 10.101.3.0 0.0.0.255
network 10.101.4.0 0.0.0.255
network 10.101.6.0 0.0.0.255
network 192.168.150.4 0.0.0.0
network x.x.x.x
passive-interface GigabitEthernet0/0/0
!
router bgp 65008
bgp log-neighbor-changes
redistribute connected
neighbor x.x.x.x remote-as 65000
!
no ip http server
ip http authentication local
no ip http secure-server
ip forward-protocol nd
ip nat inside source static 10.13.4.201 10.2.3.103
ip nat inside source static 10.12.4.201 192.168.3.101
ip nat inside source static 10.12.4.202 192.168.3.102
ip nat inside source static 10.12.11.201 192.168.3.104
ip route 0.0.0.0 0.0.0.0 Cellular0/2/0 254
ip route 10.11.4.101 255.255.255.255 10.13.200.13
ip route 10.12.4.0 255.255.255.0 10.12.200.13
ip route 10.12.11.0 255.255.255.0 10.12.200.13
ip route 10.13.4.0 255.255.255.0 10.13.200.13
ip route 10.101.1.253 255.255.255.255 152.192.63.142 254
ip route 192.168.1.220 255.255.255.255 Cellular0/2/0
ip route 192.168.7.220 255.255.255.255 Cellular0/2/0
ip tacacs source-interface Loopback0
ip ssh rsa keypair-name NAME-KEY
ip ssh version 2
!
!
ip access-list standard Member-Routes
10 permit 192.168.150.4
20 permit XX.XX.XX.XX 0.0.0.255
ip access-list standard eigrp-out
10 permit 192.168.250.152
20 permit 192.168.251.38
30 permit XX.XX.XX.XX 0.0.0.255
ip access-list standard eigrp-out-dr
10 permit 192.168.251.38
20 permit XX.XX.XX.XX 0.0.0.255
ip access-list standard telnet-access
10 permit 192.168.3.19
20 permit 192.168.1.0 0.0.0.255
30 permit 10.254.254.0 0.0.0.255
40 permit 192.168.10.0 0.0.0.255
50 permit 192.168.250.0 0.0.0.255
60 permit 192.168.9.0 0.0.0.255
70 permit 192.168.16.0 0.0.0.255
80 permit 192.168.7.0 0.0.0.255
90 permit 172.16.32.0 0.0.0.255
ip access-list standard valid-routes-eigrp
10 permit 192.168.250.152
20 permit 192.168.251.38
30 permit XX.XX.XX.XX 0.0.0.255
!
ip access-list extended Block-YYZ
10 deny ip host 192.168.3.50 any
20 deny icmp host 192.168.3.50 any
30 permit ip any any
ip access-list extended CHADunity-Access
10 remark ** CHADunity ACL XX.XX.XX.XX 0.0.0.255 is LAN IP Subnet ***
10 remark ** ICCP uses port 102 ***
10 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.3.0 0.0.0.255 eq 102
20 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.4.0 0.0.0.255 eq 102
30 permit tcp XX.XX.XX.XX 0.0.0.255 eq 102 192.168.3.0 0.0.0.255
40 permit tcp XX.XX.XX.XX 0.0.0.255 eq 102 192.168.4.0 0.0.0.255
50 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.103.0 0.0.0.255 eq 102
60 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.104.0 0.0.0.255 eq 102
70 permit tcp XX.XX.XX.XX 0.0.0.255 eq 102 192.168.103.0 0.0.0.255
80 permit tcp XX.XX.XX.XX 0.0.0.255 eq 102 192.168.104.0 0.0.0.255
90 remark ** XXX uses port 20000 ***
90 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.3.0 0.0.0.255 eq 20000
100 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.4.0 0.0.0.255 eq 20000
110 permit tcp XX.XX.XX.XX 0.0.0.255 eq 20000 192.168.3.0 0.0.0.255
120 permit tcp XX.XX.XX.XX 0.0.0.255 eq 20000 192.168.4.0 0.0.0.255
130 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.103.0 0.0.0.255 eq 20000
140 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.104.0 0.0.0.255 eq 20000
150 permit tcp XX.XX.XX.XX 0.0.0.255 eq 20000 192.168.103.0 0.0.0.255
160 permit tcp XX.XX.XX.XX 0.0.0.255 eq 20000 192.168.104.0 0.0.0.255
170 permit udp XX.XX.XX.XX 0.0.0.255 192.168.3.0 0.0.0.255 eq 20000
180 permit udp XX.XX.XX.XX 0.0.0.255 192.168.4.0 0.0.0.255 eq 20000
190 permit udp XX.XX.XX.XX 0.0.0.255 eq 20000 192.168.3.0 0.0.0.255
200 permit udp XX.XX.XX.XX 0.0.0.255 eq 20000 192.168.4.0 0.0.0.255
210 permit udp XX.XX.XX.XX 0.0.0.255 192.168.103.0 0.0.0.255 eq 20000
220 permit udp XX.XX.XX.XX 0.0.0.255 192.168.104.0 0.0.0.255 eq 20000
230 permit udp XX.XX.XX.XX 0.0.0.255 eq 20000 192.168.103.0 0.0.0.255
240 permit udp XX.XX.XX.XX 0.0.0.255 eq 20000 192.168.104.0 0.0.0.255
250 remark ** WWW web traffic to AADSC ***
250 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.3.0 0.0.0.255 eq www
260 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.4.0 0.0.0.255 eq www
270 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.103.0 0.0.0.255 eq www
280 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.104.0 0.0.0.255 eq www
290 remark ** FTP access to NAME ftp server ***
290 permit tcp XX.XX.XX.XX 0.0.0.255 host 192.168.2.3 eq ftp
300 remark ** Allow return traffic for permitted applications **
300 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.3.0 0.0.0.255 established
310 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.4.0 0.0.0.255 established
320 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.103.0 0.0.0.255 established
330 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.104.0 0.0.0.255 established
340 permit tcp XX.XX.XX.XX 0.0.0.255 host 192.168.2.3 established
350 remark ** PERMIT ICMP to AADSC Networks *** 0.0.0.255 any unreachable
350 permit icmp XX.XX.XX.XX 0.0.0.255 any time-exceeded
360 permit icmp XX.XX.XX.XX 0.0.0.255 any ttl-exceeded
370 remark ** deny
370 remark ** PERMIT ICMP to AADSC Networks ****
370 permit icmp XX.XX.XX.XX 0.0.0.255 192.168.3.0 0.0.0.255 echo
380 permit icmp XX.XX.XX.XX 0.0.0.255 192.168.4.0 0.0.0.255 echo
390 permit icmp XX.XX.XX.XX 0.0.0.255 192.168.103.0 0.0.0.255 echo
400 permit icmp XX.XX.XX.XX 0.0.0.255 192.168.104.0 0.0.0.255 echo
410 permit icmp XX.XX.XX.XX 0.0.0.255 any echo-reply
420 permit icmp XX.XX.XX.XX 0.0.0.255 any unreachable
430 remark ** deny Microsoft Netbios stuff
430 deny tcp any any range msrpc 139
440 deny udp any any range 135 netbios-ss
450 remark ** deny Multicast
450 deny ip any 224.0.0.0 31.255.255.255
460 remark ** Permit PING to Router interface **
460 permit icmp XX.XX.XX.XX 0.0.0.255 XX.XX.XX.XX 0.0.0.255 echo
470 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.3.0 0.0.0.255 eq 399 log
480 permit tcp XX.XX.XX.XX 0.0.0.255 192.168.4.0 0.0.0.255 eq 399 log
490 permit tcp XX.XX.XX.XX 0.0.0.255 eq 399 192.168.3.0 0.0.0.255 log
500 permit tcp XX.XX.XX.XX 0.0.0.255 eq 399 192.168.4.0 0.0.0.255 log
510 permit tcp XX.XX.XX.XX 0.0.0.255 10.2.3.0 0.0.0.255 eq 102
520 permit tcp XX.XX.XX.XX 0.0.0.255 eq 102 10.2.3.0 0.0.0.255
530 permit tcp XX.XX.XX.XX 0.0.0.255 10.2.3.0 0.0.0.255 eq 20000
540 permit tcp XX.XX.XX.XX 0.0.0.255 eq 20000 10.2.3.0 0.0.0.255
550 permit tcp XX.XX.XX.XX 0.0.0.255 10.2.3.0 0.0.0.255 established
560 permit icmp XX.XX.XX.XX 0.0.0.255 10.2.3.0 0.0.0.255 echo
570 permit icmp host 194.49.37.80 host 10.2.3.103
580 permit udp host 192.168.15.50 any eq ntp
590 permit tcp XX.XX.XX.XX 0.0.0.255 10.12.4.0 0.0.0.255 eq 102
600 permit tcp XX.XX.XX.XX 0.0.0.255 eq 102 10.12.4.0 0.0.0.255
610 permit tcp XX.XX.XX.XX 0.0.0.255 10.12.4.0 0.0.0.255 established
620 permit icmp XX.XX.XX.XX 0.0.0.255 10.12.4.0 0.0.0.255
630 permit tcp XX.XX.XX.XX 0.0.0.255 10.11.4.0 0.0.0.255 eq 102
640 permit tcp XX.XX.XX.XX 0.0.0.255 eq 102 10.11.4.0 0.0.0.255
650 permit tcp XX.XX.XX.XX 0.0.0.255 10.11.4.0 0.0.0.255 established
660 permit icmp XX.XX.XX.XX 0.0.0.255 10.11.4.0 0.0.0.255
670 permit tcp XX.XX.XX.XX 0.0.0.255 10.12.11.0 0.0.0.255 eq 102
680 permit tcp XX.XX.XX.XX 0.0.0.255 eq 102 10.12.11.0 0.0.0.255
690 permit tcp XX.XX.XX.XX 0.0.0.255 10.12.11.0 0.0.0.255 established
700 permit icmp XX.XX.XX.XX 0.0.0.255 10.12.11.0 0.0.0.255
ip access-list extended DMVPN
10 permit eigrp any any
11 permit ip any any
20 deny ip any any
--> ip access-list extended GREINIPSEC
ip access-list extended AADSC
10 permit tcp any 192.168.3.0 0.0.0.255 eq 102
20 permit tcp any 192.168.4.0 0.0.0.255 eq 102
!
map-class frame-relay FR-56K-with-policy
frame-relay bc 560
logging host 192.168.1.146
ip access-list standard 10
10 remark Server we obtain time from
10 permit 192.168.15.50
20 deny any
ip access-list standard 91
10 permit 192.168.3.19
20 permit 192.168.1.146
ip access-list extended 100
10 permit gre host 10.101.1.8 host 10.101.1.1
ip access-list extended 102
10 remark Define interesting traffic for dialer
10 deny eigrp any any
20 permit ip any any
ip access-list extended 110
10 permit gre host 10.101.1.8 host 10.101.1.254
ip access-list extended 120
10 permit gre host 10.101.1.8 host 10.101.1.253
ip access-list extended 130
10 permit ip host 194.49.37.80 host 10.12.4.101
20 permit ip host 194.49.37.99 host 10.12.4.101
30 permit ip host 194.49.37.80 host 10.12.4.102
40 permit ip host 194.49.37.99 host 10.12.4.102
ip access-list extended 140
10 permit ip host 194.49.37.80 host 10.11.4.101
20 permit ip host 194.49.37.99 host 10.11.4.101
dialer watch-list 1 ip 192.168.3.0 255.255.255.0
dialer watch-list 1 ip 192.168.4.0 255.255.255.0
dialer watch-list 1 ip 10.2.3.0 255.255.255.0
dialer-list 1 protocol ip permit
!
snmp-server CHADunity XXXXXXXX RO 91
snmp-server location CHADunity fire domainerative Windsor, VA
snmp-server enable traps snmp authentication coldstart warmstart
!
tacacs-server key 7 044804071F25455D01
tacacs server 192.168.1.163
address ipv4 192.168.1.163
tacacs server 10.2.1.95
address ipv4 10.2.1.95
timeout 20
!
control-plane
!
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default
!
line con 0
password 7 11584B56
stopbits 1
line aux 0
line vty 0 4
password 7 13544541
logging synchronous
transport input ssh
line vty 5 15
password 7 13544541
logging synchronous
transport input ssh
line vty 16 96
transport input ssh
line vty 97
no exec
transport input ssh
!
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http
ntp authenticate
ntp trusted-key 5
ntp source Loopback0
ntp access-group peer 10
ntp server 192.168.15.50
!
end

[KMNRuser]

Georg,

Yes i had removed it at some point.

It is back in place now..

ip access-list extended DMVPN
10 permit eigrp any any
11 permit ip any any
20 deny ip any any
ip access-list extended GREINIPSEC
10 permit gre any any
ip access-list extended SCADA
10 permit tcp any 192.168.3.0 0.0.0.255 eq 102
20 permit tcp any 192.168.4.0 0.0.0.255 eq 102
!
!
map-class frame-relay FR-56K-with-policy
frame-relay bc 560
logging host 192.168.1.146
ip access-list standard 10
10 remark Server we obtain time from
10 permit 192.168.15.50
20 deny any

So i have set up persistent logging, but what i have not yet been able to capture is that is happening on the box when i apply the "match address GREINIPSEC" command into the specific sequence of the crypto map.  I do not have the option of pulling the crypto map off of the interface, as our connectivity to the remote box depends on the other tunnels that are part of the cry map.  So each time i try to bring this tunnel up, i do a "reload in XX" command, choose NOT To save the config, and then hit enter.  then i buil the cry map sequence over, which entails setting the peer, setting the transfrom set, and then finally using the match address statement with the ACL entitled "GREINIPSEC".  When the match address statement is applied, then we loose connectivity and wait for the router to reboot, which strips back off the pieces within the sequence number within the crypto map.

I am still trying to capture using the persistent logging, but i do not see output in the logs once examined that points to where everything drops once the match address statement is applied.  And i have logging set to debugging, as well as having several cry isa and cry ipsec debugs running..

[Richard Burts]

In a previous post I made a point about the debug output indicates that the tunnel is seeing ISAKMP packets rather than GRE packets. Please add an entry in the acl that permits ISAKMP.

[KMNRuser]

Rick,

I did go back and add an ACE for the ACL entitled GREINIPSEC.  it now looks like this on both sides:

ip access-list extended GREINIPSEC
10 permit gre any any
20 permit udp any any eq isakmp

Unfortunately this did not resolve the issue; once again the connectivity to the box drops the second we configure "match address GREINIPSEC" into the Crypto map sequence.

[Richard Burts]

Thanks for adding isakmp to the acl. Sorry that it did not resolve the issue. Am I correct in understanding that the tunnel you are having trouble with is tunnel 6? Would you post what you are attempting to use to configure that tunnel, and as a point of comparison would you post the config of a tunnel that works well?

[KMNRuser]

Rick,

Yes you are correct..Tunnel 6 is the ipsec-isakmp tunnel I am trying to bring up.

Here is the config of the tunnel at the HUB.  This box is an 8200 router.

interface Tunnel6
ip address 10.101.6.1 255.255.255.0
tunnel source GigabitEthernet0/0/0
tunnel destination 10.140.0.20

crypto isakmp policy 10
encryption aes
hash sha256
authentication pre-share
group 14

crypto isakmp key 6 l~Dr2OobK1=1y#k address 10.140.0.20

crypto ipsec transform-set CommunityTS esp-aes 256 esp-sha512-hmac
mode tunnel

crypto isakmp keepalive 10

crypto map S2S 20 ipsec-isakmp
set peer 10.140.0.20
set transform-set CommunityTS
match address GREINIPSEC

It is applied on our G0/0/0 interface facing our Verizon MPLS:

interface GigabitEthernet0/0/0
description Verizon-C1178910
ip address 123.123.248.46 255.255.255.252
ip nat outside
speed 100
no negotiation auto
crypto map S2S

Here is the config of the tunnel at the remote end.  This is an 1100 Series industrial router.

interface Tunnel6
ip address 10.101.6.2 255.255.255.0
tunnel source Cellular0/2/0
tunnel destination 123.123.248.46

crypto isakmp policy 10
encryption aes
hash sha256
authentication pre-share
group 14

crypto isakmp key 6 l~Dr2OobK1=1y#k address 123.123.248.46

crypto ipsec transform-set CommunityTS esp-aes 256 esp-sha512-hmac
mode tunnel

There is an existing crypto map entitled "ICCP_BACKUP" attached to the Cellular 0/2/0 interface.  This is the crypto map that i had originally eluded to that i thought had to be removed from the interface prior to manipulation (there is still part of me that thinks this is the issue). This is the contents of the crypto map as it exists PRIOR to any manipulation that i do:

crypto map ICCP_BACKUP 50 ipsec-isakmp
set peer 152.162.188.206
set transform-set ODEC_WAN
match address 100
crypto map ICCP_BACKUP 70 ipsec-isakmp
set peer 152.164.249.218
set transform-set ODEC_WAN
match address 100
crypto map ICCP_BACKUP 80 ipsec-isakmp
set peer 152.176.127.14
set transform-set ODEC_WAN
match address 110
crypto map ICCP_BACKUP 90 ipsec-isakmp
set peer 152.185.254.26
set transform-set ODEC_WAN
match address 120

I have tried to insert a new sequence number at all places within this crypto map.. at the beginning, after the 1st, 2nd, and 3rd entries, and at the end.  So far it does not seem to matter because irrespective of where i place the following config it fails, we loose connectivity, and only by virtue of the reload command, which we set each time before we start working, do we get the router back:

Crypto map ICCP_BACKUP 91 ipsec-isakmp

set peer 123.123.248.46

set transform-set CommunityTS

match address GREINIPSEC

Yesterday using extended logging i was able to piece together a log file with crypto debugs running when we applied the "match address GREINIPSEC" statement to the crypto map.  I can make that available for you if you think it would be helpful.

Thank you!

KMNRuser