Solved: Re: IPSEC Tunnel Fails 2x2921 - Please Help - Page 2

clyde.a.huffman.ctr@mail.mil · ‎12-14-2018

I tried putting a routing statement but no change. NO PRIVATE INFO: I'll change the crypto key once I get this working.

ip route 192.168.175.0 255.255.255.0 192.168.176.1

ip route 192.168.176.0 255.255.255.0 192.168.175.1

!!!@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ SITE 1 @@@@@@@@@@@@@@@@@@@@@@@@!!!!
localrtr#sh run
hostname localrtr
boot-start-marker
boot-end-marker
enable secret 5 $1$A3Kg$TZeqZI6QF3r.S4nu80fZJ1
no aaa new-model
!
ip domain name mydomain.com
ip cef
!
multilink bundle-name authenticated
username cisco privilege 0 password 7 05190900355E41060D
!
crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp key firewallcx address 192.168.168.236
!
crypto ipsec transform-set TS esp-3des esp-md5-hmac
!
crypto map CMAP 10 ipsec-isakmp
set peer 192.168.168.236
set transform-set TS
match address VPN_TRAFFIC
!
interface GigabitEthernet0/0
description OUTSIDE
ip address 192.168.168.235 255.255.255.0
duplex auto
speed auto
crypto map CMAP
!
interface GigabitEthernet0/1
description INSIDE
ip address 192.168.175.1 255.255.255.0
duplex auto
speed auto
!
interface GigabitEthernet0/2
no ip address
duplex auto
speed auto
!
interface FastEthernet0/0/0
description MGNT_10_10_10_15
switchport access vlan 200
no ip address
!
interface Vlan200
ip address 10.10.10.15 255.255.255.224
!
ip access-list extended VPN_TRAFFIC
permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255
!
control-plane
!
line con 0
line vty 0 4
login local
transport input ssh
!
end
---------------------================================-----------------------------
localrtr#ping 192.168.176.1 so gi0/1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
.....
Success rate is 0 percent (0/5)
localrtr#
localrtr#ping 192.168.168.236
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.168.236, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
localrtr#

---------------------================================-----------------------------
localrtr# debug crypto cond peer ipv4 192.168.176.1
localrtr# debug crypto ipsec
localrtr# debug crypto isakmp
localrtr# term mon

localrtr# sh cry isa pol

Global IKE policy
Protection suite of priority 1
   encryption algorithm:   Three key triple DES
   hash algorithm:       Message Digest 5
   authentication method:   Pre-Shared Key
   Diffie-Hellman group:   #2 (1024 bit)
   lifetime:       86400 seconds, no volume limit
localrtr#
localrtr#sh crypto map
Crypto Map IPv4 "CMAP" 10 ipsec-isakmp
   Peer = 192.168.168.236
   Extended IP access list VPN_TRAFFIC
        access-list VPN_TRAFFIC permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255
   Current peer: 192.168.168.236
   Security association lifetime: 4608000 kilobytes/3600 seconds
   Responder-Only (Y/N): N
   PFS (Y/N): N
   Transform sets={
       TS: { esp-3des esp-md5-hmac } ,
   }
   Interfaces using crypto map CMAP:
       GigabitEthernet0/0

localrtr#

localrtr#sh crypto session
Crypto session current status

Interface: GigabitEthernet0/0
Session status: DOWN
Peer: 192.168.168.236 port 500 localrtr#sh crypto session
Crypto session current status

Interface: GigabitEthernet0/0
Session status: DOWN
Peer: 192.168.168.236 port 500
IPSEC FLOW: permit ip 192.168.175.0/255.255.255.0 192.168.176.0/255.255.255.0
        Active SAs: 0, origin: crypto map

localrtr#
Dec 14 17:43:46.005: No peer struct to get peer description
localrtr#

---------------------------==========================------------------------
!!!@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ SITE 2 @@@@@@@@@@@@@@@@@@@@@@@@@!!!!
remotertr#sh run
hostname remotertr
boot-start-marker
boot-end-marker
enable secret 5 $1$m3qS$tiNd8YH.rmhKzGoRqa2970
no aaa new-model
!
ip domain name mydomain.com
!
multilink bundle-name authenticated
username cisco privilege 0 password 7 105C061611051D0418
!
crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp key firewallcx address 192.168.168.235
!
crypto ipsec transform-set TS esp-3des esp-md5-hmac
mode tunnel
!
crypto map CMAP 10 ipsec-isakmp
set peer 192.168.168.235
set transform-set TS
match address VPN_TRAFFIC
!
interface GigabitEthernet0/0
description OUTSIDE
ip address 192.168.168.236 255.255.255.0
duplex auto
speed auto
crypto map CMAP
!
interface GigabitEthernet0/1
description INSIDE
ip address 192.168.176.1 255.255.255.0
duplex auto
speed auto
!
interface GigabitEthernet0/2
description MNGT
ip address 10.10.10.16 255.255.255.224
duplex auto
speed auto
!
ip access-list extended VPN_TRAFFIC
permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255
!
control-plane
!
line con 0
line vty 0 4
login local
transport input ssh
!
end

-----------------------------======================-----------------------
remotertr#ping 192.168.175.1 so gi0/1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.175.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.176.1
.....
Success rate is 0 percent (0/5)
remotertr#
remotertr#ping 192.168.168.235
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.168.235, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
remotertr#

-----------------------------======================-----------------------
remotertr#debug crypto cond peer ipv4 192.168.175.1
remotertr#debug crypto ipsec
remotertr#debug crypto isakmp
remotertr#term mon

remotertr#sh cry isa pol

Global IKE policy
Protection suite of priority 1
   encryption algorithm:   Three key triple DES
   hash algorithm:       Message Digest 5
   authentication method:   Pre-Shared Key
   Diffie-Hellman group:   #2 (1024 bit)
   lifetime:       86400 seconds, no volume limit
remotertr#
remotertr#sh crypto map
Crypto Map IPv4 "CMAP" 10 ipsec-isakmp
   Peer = 192.168.168.235
   Extended IP access list VPN_TRAFFIC
        access-list VPN_TRAFFIC permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255
   Current peer: 192.168.168.235
   Security association lifetime: 4608000 kilobytes/3600 seconds
   Responder-Only (Y/N): N
   PFS (Y/N): N
   Mixed-mode : Disabled
   Transform sets={
       TS: { esp-3des esp-md5-hmac } ,
   }
   Interfaces using crypto map CMAP:
       GigabitEthernet0/0

   Interfaces using crypto map NiStTeSt1:

remotertr#

remotertr#sh crypto map
Crypto Map IPv4 "CMAP" 10 ipsec-isakmp
   Peer = 192.168.168.235
   Extended IP access list VPN_TRAFFIC
        access-list VPN_TRAFFIC permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255
   Current peer: 192.168.168.235
   Security association lifetime: 4608000 kilobytes/3600 seconds
   Responder-Only (Y/N): N
   PFS (Y/N): N
   Mixed-mode : Disabled
   Transform sets={
       TS: { esp-3des esp-md5-hmac } ,
   }
   Interfaces using crypto map CMAP:
       GigabitEthernet0/0

   Interfaces using crypto map NiStTeSt1:

remotertr#

remotertr#sh crypto session
Crypto session current status

Interface: GigabitEthernet0/0
Session status: DOWN
Peer: 192.168.168.235 port 500
IPSEC FLOW: permit ip 192.168.176.0/255.255.255.0 192.168.175.0/255.255.255.0
        Active SAs: 0, origin: crypto map

remotertr#
*Dec 14 17:18:13.423: No peer struct to get peer description
remotertr#

---------------------========================-----------------------

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

Let's start from the top. I thought that is might be IOS related, so I copied both IOS's to each router and tested both. Same results. The tunnel will not come up until I put "any" as the CRYPTOMAP destination in the ACL. The R1 ACL shows matches but none on R2. ?????????????

-------------------------------R1---------------------------------------------

crypto isakmp policy 1

encr 3des

hash md5

authentication pre-share

group 2

lifetime 86400
!

crypto isakmp key firewallcx address 192.168.168.236

!

crypto ipsec transform-set TS esp-3des esp-md5-hmac

mode tunnel

!

ip access-list extended VPN_TRAFFIC

permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255

!

crypto map CMAP 10 ipsec-isakmp

set peer 192.168.168.236

set transform-set TS

match address VPN_TRAFFIC
!

interface GigabitEthernet0/0

description OUTSIDE

ip address 192.168.168.235 255.255.255.0

crypto map CMAP

!

interface GigabitEthernet0/1

description INSIDE

ip address 192.168.175.1 255.255.255.0

System image file is "flash:c2900-universalk9-mz.SPA.154-3.M1.bin"

------------------------------R2---------------------------------------------

crypto isakmp policy 1

encr 3des

hash md5

authentication pre-share

group 2

lifetime 86400
!

crypto isakmp key firewallcx address 192.168.168.235
!

crypto ipsec transform-set TS esp-3des esp-md5-hmac

mode tunnel

!

ip access-list extended VPN_TRAFFIC

permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255

!

crypto map CMAP 10 ipsec-isakmp

set peer 192.168.168.235

set transform-set TS

match address VPN_TRAFFIC

!

interface GigabitEthernet0/0

description OUTSIDE

ip address 192.168.168.236 255.255.255.0

crypto map CMAP

!

interface GigabitEthernet0/1

description INSIDE

ip address 192.168.176.1 255.255.255.0

System image file is "flash:c2900-universalk9-mz.SPA.154-3.M1.bin"

-----------------------------TUNNEL DOWN------------------------------

localrtr#clear crypto isakmp

localrtr#clear crypto sa

remotertr#clear crypto isakmp

remotertr#clear crypto sa

localrtr#sh cry session
Crypto session current status

Interface: GigabitEthernet0/0
Session status: DOWN-NEGOTIATING
Peer: 192.168.168.236 port 500
Session ID: 0
IKEv1 SA: local 192.168.168.235/500 remote 192.168.168.236/500 Inactive
IPSEC FLOW: permit ip 192.168.175.0/255.255.255.0 192.168.176.0/255.255.255.0
Active SAs: 0, origin: crypto map

localrtr#sh cry session
Crypto session current status

Interface: GigabitEthernet0/0
Session status: DOWN
Peer: 192.168.168.236 port 500
IPSEC FLOW: permit ip 192.168.175.0/255.255.255.0 192.168.176.0/255.255.255.0
Active SAs: 0, origin: crypto map

localrtr#

-----------------------------pings with tunnel down--------------------------

localrtr#ping 191.168.176.1 so 192.168.175.1
Sending 5, 100-byte ICMP Echos to 191.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
.....

remotertr#ping 191.168.175.1 so 192.168.176.1
Sending 5, 100-byte ICMP Echos to 191.168.175.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.176.1
.....

-----------------------------TUNNEL COMING UP-------------------------------

NOW do something really weird. Add any to the CRYTOMAP and the tunnel will come UP

localrtr(config)# ip access-list extended VPN_TRAFFIC
20 permit ip 192.168.175.0 0.0.0.255 any

remotertr(config)# ip access-list extended VPN_TRAFFIC

20 permit ip 192.168.176.0 0.0.0.255 any

- - - - - - - - - WEIRD - - - - - - - - -

localrtr(config)#ip access-list extended VPN_TRAFFIC
localrtr(config-ext-nacl)#20 permit ip 192.168.175.0 0.0.0.255 any
localrtr(config-ext-nacl)#do sh cry ses
Crypto session current status

Interface: GigabitEthernet0/0
Session status: UP-IDLE
Peer: 192.168.168.236 port 500
Session ID: 0
IKEv1 SA: local 192.168.168.235/500 remote 192.168.168.236/500 Active
IPSEC FLOW: permit ip 192.168.175.0/255.255.255.0 192.168.176.0/255.255.255.0
Active SAs: 0, origin: crypto map
IPSEC FLOW: permit ip 192.168.175.0/255.255.255.0 0.0.0.0/0.0.0.0
Active SAs: 0, origin: crypto map

localrtr(config-ext-nacl)#

remotertr#conf t
Enter configuration commands, one per line. End with CNTL/Z.
remotertr(config)#ip access-list extended VPN_TRAFFIC
remotertr(config-ext-nacl)#20 permit ip 192.168.176.0 0.0.0.255 any
remotertr(config-ext-nacl)#do sh cry ses
Crypto session current status

Interface: GigabitEthernet0/0
Session status: UP-IDLE
Peer: 192.168.168.235 port 500
Session ID: 0
IKEv1 SA: local 192.168.168.236/500 remote 192.168.168.235/500 Active
IPSEC FLOW: permit ip 192.168.176.0/255.255.255.0 192.168.175.0/255.255.255.0

Active SAs: 0, origin: crypto map
IPSEC FLOW: permit ip 192.168.176.0/255.255.255.0 0.0.0.0/0.0.0.0
Active SAs: 0, origin: crypto map

remotertr(config-ext-nacl)#

-------------------------------ACL MATCHEDS------------------------------

localrtr(config-ext-nacl)#do sh ip access-l
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255 (1516 matches)
20 permit ip 192.168.175.0 0.0.0.255 any (97 matches)
localrtr(config-ext-nacl)#

remotertr(config-ext-nacl)#do sh ip access-list
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255
20 permit ip 192.168.176.0 0.0.0.255 any
remotertr(config-ext-nacl)#

---------------------------SH LICENSES-------------------------

R1 ------------------------------------------------------------------------
ipbase ipbasek9 Permanent ipbasek9
security securityk9 Permanent securityk9
uc uck9 Permanent uck9
data datak9 Permanent datak9

R2 ------------------------------------------------------------------------
ipbase ipbasek9 Permanent ipbasek9
security securityk9 Permanent securityk9
uc uck9 Permanent uck9

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

-------------------------SH CRYPTO IPSEC SA----------------------------------

localrtr(config-ext-nacl)#ip access-list extended VPN_TRAFFIC
localrtr(config-ext-nacl)#no 20 permit ip 192.168.175.0 0.0.0.255 any

remotertr(config-ext-nacl)#ip access-list extended VPN_TRAFFIC
remotertr(config-ext-nacl)#no 20 permit ip 192.168.176.0 0.0.0.255 any

localrtr(config-ext-nacl)#do clear cry isa
localrtr(config-ext-nacl)#do clear cry sa
remotertr(config-ext-nacl)#do clear cry isa
remotertr(config-ext-nacl)#do clear cry sa

------------------------------SH CRY IPSEC SA--------------------------------

----------------------------R1------------------------------

localrtr#sh cry ipsec sa

interface: GigabitEthernet0/0
Crypto map tag: CMAP, local addr 192.168.168.235

protected vrf: (none)
local ident (addr/mask/prot/port): (192.168.175.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.176.0/255.255.255.0/0/0)
current_peer 192.168.168.236 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 192.168.168.235, remote crypto endpt.: 192.168.168.236
plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none

---------------------------------R2-----------------------------

remotertr#sh cry ipsec sa

interface: GigabitEthernet0/0
Crypto map tag: CMAP, local addr 192.168.168.236

protected vrf: (none)
local ident (addr/mask/prot/port): (192.168.176.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.175.0/255.255.255.0/0/0)
current_peer 192.168.168.235 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 192.168.168.236, remote crypto endpt.: 192.168.168.235
plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none

Richard Burts · ‎12-17-2018

Thanks for the additional information. This is quite surprising that adding the any parameter in the acl makes the tunnel come up. Would you post the output of show crypto ipsec sa from both routers at a time that the tunnel is up (when the acl includes the any parameter).

Is it possible that address translation is configured on either (or both) routers? If so would you post that part of the configs?

HTH

Rick

HTH

Rick

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#sh run
Building configuration...

Current configuration : 2697 bytes
!
! Last configuration change at 16:57:27 UTC Mon Dec 17 2018 by cisco
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname localrtr
!
boot-start-marker
boot system flash:c2900-universalk9-mz.SPA.154-3.M1.bin
boot-end-marker
!
!
enable secret 5 $1$A3Kg$TZeqZI6QF3r.S4nu80fZJ1
!
no aaa new-model
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
no ip domain lookup
ip domain name cortana.com
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
cts logging verbose
!
!
voice-card 0
!
!
!
!
!
!
!
!
license udi pid CISCO2921/K9 sn FTX1714AKGS
hw-module pvdm 0/0
!
!
!
username cisco privilege 0 password 7 05190900355E41060D
!
redundancy
!
!
!
!
!
ip ftp username ahuffman
ip ftp password 7 091D1C180E040408134D241B1C051B090D
!
!
crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp key firewallcx address 192.168.168.236
!
!
crypto ipsec transform-set TS esp-3des esp-md5-hmac
mode tunnel
!
!
!
crypto map CMAP 10 ipsec-isakmp
set peer 192.168.168.236
set transform-set TS
match address VPN_TRAFFIC
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
service-module enable
!
interface GigabitEthernet0/0
description OUTSIDE
ip address 192.168.168.235 255.255.255.0
duplex auto
speed auto
crypto map CMAP
!
interface GigabitEthernet0/1
description INSIDE
ip address 192.168.175.1 255.255.255.0
duplex auto
speed auto
!
interface GigabitEthernet0/2
no ip address
duplex auto
speed auto
!
interface FastEthernet0/0/0
description MGNT_10_10_10_15
switchport access vlan 200
no ip address
!
interface FastEthernet0/0/1
no ip address
!
interface FastEthernet0/0/2
no ip address
!
interface FastEthernet0/0/3
no ip address
!
interface Vlan1
no ip address
!
interface Vlan200
ip address 10.10.10.15 255.255.255.224
!
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip route 192.168.175.0 255.255.255.0 192.168.168.236
!
ip access-list extended VPN_TRAFFIC
permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255
permit ip 192.168.175.0 0.0.0.255 any
!
!
!
!
control-plane
!
!
!
!
!
!
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default
!
!
!
!
!
!
!
gatekeeper
shutdown
!
!
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
login local
transport input ssh
!
scheduler allocate 20000 1000
ntp server 192.168.168.200
!
end

localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#sh run
Building configuration...

Current configuration : 2302 bytes
!
! Last configuration change at 16:31:38 UTC Mon Dec 17 2018 by cisco
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname remotertr
!
boot-start-marker
boot system flash:c2900-universalk9-mz.SPA.154-3.M1.bin
boot-end-marker
!
!
enable secret 5 $1$m3qS$tiNd8YH.rmhKzGoRqa2970
!
no aaa new-model
!
!
!
!
!
!
!
!
!
!
!
!
!
!
no ip domain lookup
ip domain name cortana.com
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
cts logging verbose
!
!
voice-card 0
!
!
!
!
!
!
!
!
license udi pid CISCO2921/K9 sn FJC1903A25P
!
!
username cisco privilege 0 password 7 105C061611051D0418
!
redundancy
!
!
!
!
!
ip ftp username ahuffman
ip ftp password 7 040A591718205F5411582526252A3F3E12
!
!
crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp key firewallcx address 192.168.168.235
!
!
crypto ipsec transform-set TS esp-3des esp-md5-hmac
mode tunnel
!
!
!
crypto map CMAP 10 ipsec-isakmp
set peer 192.168.168.235
set transform-set TS
match address VPN_TRAFFIC
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description OUTSIDE
ip address 192.168.168.236 255.255.255.0
duplex auto
speed auto
crypto map CMAP
!
interface GigabitEthernet0/1
description INSIDE
ip address 192.168.176.1 255.255.255.0
duplex auto
speed auto
!
interface GigabitEthernet0/2
description MNGT
ip address 10.10.10.16 255.255.255.224
duplex auto
speed auto
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip route 192.168.176.0 255.255.255.0 192.168.168.235
!
ip access-list extended VPN_TRAFFIC
permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255
permit ip 192.168.176.0 0.0.0.255 any
!
!
!
!
control-plane
!
!
!
!
!
!
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default
!
!
!
!
!
!
!
gatekeeper
shutdown
!
!
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
login local
transport input ssh
!
scheduler allocate 20000 1000
!
end

remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#
remotertr#

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

-----------------------------------------R1-------------------------------------
localrtr(config-ext-nacl)#
localrtr(config-ext-nacl)#
localrtr(config-ext-nacl)#
localrtr(config-ext-nacl)#do sh ip access-l
Extended IP access list VPN_TRAFFIC
    10 permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255 (1032 matches)
    20 permit ip 192.168.175.0 0.0.0.255 any (313 matches)
localrtr(config-ext-nacl)#do sh cry ses
Crypto session current status

Interface: GigabitEthernet0/0
Session status: UP-IDLE
Peer: 192.168.168.236 port 500
Session ID: 0
IKEv1 SA: local 192.168.168.235/500 remote 192.168.168.236/500 Active
IPSEC FLOW: permit ip 192.168.175.0/255.255.255.0 192.168.176.0/255.255.255.0
        Active SAs: 0, origin: crypto map
IPSEC FLOW: permit ip 192.168.175.0/255.255.255.0 0.0.0.0/0.0.0.0
        Active SAs: 0, origin: crypto map

localrtr(config-ext-nacl)#do sh cry ipsec sa

interface: GigabitEthernet0/0
    Crypto map tag: CMAP, local addr 192.168.168.235

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.175.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.176.0/255.255.255.0/0/0)
   current_peer 192.168.168.236 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 192.168.168.235, remote crypto endpt.: 192.168.168.236
     plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
     current outbound spi: 0x0(0)
     PFS (Y/N): N, DH group: none

     inbound esp sas:

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.175.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer 192.168.168.236 port 500
     PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 320, #recv errors 0

     local crypto endpt.: 192.168.168.235, remote crypto endpt.: 192.168.168.236
     plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
     current outbound spi: 0x0(0)
     PFS (Y/N): N, DH group: none

     inbound esp sas:

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:
localrtr(config-ext-nacl)#
localrtr(config-ext-nacl)#
localrtr(config-ext-nacl)#
localrtr(config-ext-nacl)#

---------------------------------------------------_R2-------------------------------------------
remotertr(config)#
remotertr(config)#
remotertr(config)#
remotertr(config)#do sh ip access-l
Extended IP access list VPN_TRAFFIC
    10 permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255
    20 permit ip 192.168.176.0 0.0.0.255 any
remotertr(config)#do sh cry ses
Crypto session current status

Interface: GigabitEthernet0/0
Session status: UP-IDLE
Peer: 192.168.168.235 port 500
Session ID: 0
IKEv1 SA: local 192.168.168.236/500 remote 192.168.168.235/500 Active
IPSEC FLOW: permit ip 192.168.176.0/255.255.255.0 192.168.175.0/255.255.255.0
        Active SAs: 0, origin: crypto map
IPSEC FLOW: permit ip 192.168.176.0/255.255.255.0 0.0.0.0/0.0.0.0
        Active SAs: 0, origin: crypto map

remotertr(config)#do sh cry ipsec sa

interface: GigabitEthernet0/0
    Crypto map tag: CMAP, local addr 192.168.168.236

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.176.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.175.0/255.255.255.0/0/0)
   current_peer 192.168.168.235 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 192.168.168.236, remote crypto endpt.: 192.168.168.235
     plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
     current outbound spi: 0x0(0)
     PFS (Y/N): N, DH group: none

     inbound esp sas:

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.176.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer 192.168.168.235 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 192.168.168.236, remote crypto endpt.: 192.168.168.235
     plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
     current outbound spi: 0x0(0)
     PFS (Y/N): N, DH group: none

     inbound esp sas:

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:
remotertr(config)#
remotertr(config)#
remotertr(config)#
remotertr(config)#

Richard Burts · ‎12-17-2018

I have re-read this thread several times and have some additional comments/questions.

I see this in some output from router 2

Interfaces using crypto map NiStTeSt1:

I do not see any reference to that crypto map in the configs and assume that it was perhaps in an earlier version of the config and has been removed. I would suggest reboot of both routers to remove any lingering memory of things that have been removed from the configs.

I noticed that when you were setting up conditional debug (which is a VERY helpful tool when troubleshooting vpn) that the address you specified (which should be the peer address) was the address of the peer lan - but not the peer address (192.168.168.235/236). If you clean that up done know if debug would have helpful information.

I do not see any route statements in the configs that you posted. And in one post you suggest that the tunnels act like routing statements and so you do not need route statements. That is not the case. To clarify the issue would you post the output of show ip route from both routers?

HTH

Rick

HTH

Rick

Richard Burts · ‎12-17-2018

I see that while I was re-reading and then writing my response that you posted current configs. They show that there is no address translation and that there is now routing statements for the remote subnets. So 2 of my theories are demonstrated to not be the issue.

I also see that you have posted the output of show crypto ipsec sa. What is interesting there is that while it does show that the tunnel is up it also shows that 0 packets encapsulated and 0 packets decaps. So even though the tunnel is up it is not carrying any traffic. So we have some issue to find beyond adding the any parameter to the acl.

HTH

Rick

HTH

Rick

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

Richard, thanks for your help. The tunnel just pinged !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Your the best!

Since it is working, can I leave the other 0.0.0.0 SA? Is it a security risk? Should I try to remove the any from the ACL?

--------------------------------ROUTE STATEMENT-----------------------------

localrtr(config)#ip route 0.0.0.0 0.0.0.0 192.168.168.236
localrtr(config)#do sh run | i NiStTeSt1
localrtr(config)#do ping 192.168.176.1 so 192.168.175.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
...!!
Success rate is 40 percent (2/5), round-trip min/avg/max = 1/1/1 ms
localrtr(config)#do ping 192.168.176.1 so 192.168.175.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
localrtr(config)#

remotertr(config)#ip route 0.0.0.0 0.0.0.0 192.168.168.235
remotertr(config)#do sh run | i NiStTeSt1
remotertr(config)#

do ping 192.168.175.1 so 192.168.176.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.175.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.176.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
remotertr(config)#

--------------------------------ACL MATCHES-------------------------------------

localrtr(config)#do sh ip access-l
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255 (1765 matches)
20 permit ip 192.168.175.0 0.0.0.255 any (645 matches)
localrtr(config)#

emotertr(config)#do sh ip access-l
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255 (12 matches)
20 permit ip 192.168.176.0 0.0.0.255 any
remotertr(config)#

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

The 0.0.0.0 is used in the R1 ACL but not in the R2 ACL? Is there something wrong with R1?

----------------------------------R1 ACL-----------------------------

localrtr#clear access-list counters
localrtr#ping 192.168.176.1 so 192.168.175.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
localrtr#sh ip access-
localrtr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255 (29 matches)
20 permit ip 192.168.175.0 0.0.0.255 any (9 matches)
localrtr#ping 192.168.176.1 so 192.168.175.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
localrtr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255 (44 matches)
20 permit ip 192.168.175.0 0.0.0.255 any (14 matches)
localrtr#ping 192.168.176.1 so 192.168.175.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
localrtr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255 (57 matches)
20 permit ip 192.168.175.0 0.0.0.255 any (18 matches)
localrtr#ping 192.168.176.1 so 192.168.175.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
localrtr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255 (78 matches)
20 permit ip 192.168.175.0 0.0.0.255 any (26 matches)
localrtr#ping 192.168.176.1 so 192.168.175.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.176.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.175.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
localrtr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255 (355 matches)
20 permit ip 192.168.175.0 0.0.0.255 any (123 matches)
localrtr#

-----------------------------------R2 ACL--------------------------------------

emotertr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255 (37 matches)
20 permit ip 192.168.176.0 0.0.0.255 any
remotertr#ping 192.168.175.1 so 192.168.176.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.175.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.176.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
remotertr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255 (42 matches)
20 permit ip 192.168.176.0 0.0.0.255 any
remotertr#ping 192.168.175.1 so 192.168.176.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.175.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.176.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
remotertr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255 (47 matches)
20 permit ip 192.168.176.0 0.0.0.255 any
remotertr#ping 192.168.175.1 so 192.168.176.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.175.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.176.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
remotertr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255 (52 matches)
20 permit ip 192.168.176.0 0.0.0.255 any
remotertr#ping 192.168.175.1 so 192.168.176.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.175.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.176.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
remotertr#sh ip access-lists
Extended IP access list VPN_TRAFFIC
10 permit ip 192.168.176.0 0.0.0.255 192.168.175.0 0.0.0.255 (57 matches)
20 permit ip 192.168.176.0 0.0.0.255 any
remotertr#

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

Did some testing, it was the routing statement all the time. I removed the "any" from the ACL and now the L2 tunnel is up and working. Thank you Richard

Richard Burts · ‎12-17-2018

I am very glad (and a bit relieved) that the problem is solved and that my suggestion pointed in the right direction. The issue with a crypto acl that uses permit any is that when applied any traffic from the source subnet going out the outside interface will be encrypted and sent through the tunnel. That would impact the ability of anything on the source subnet to access resources in the Internet.

Thank you for marking this question as solved. This will help other participants in the community to identify discussions that have helpful information. This has been an extensive discussion with an interesting outcome. I believe other participants will benefit from reading it.

HTH

Rick

HTH

Rick

clyde.a.huffman.ctr@mail.mil · ‎12-17-2018

Hi Richard, I did turn every color getting this fixed - thank you again for your help. I seem to have forgotten that a routing statement is needed :-/

I've put in specific routing statements into the configs so now I can move ahead and add the overload NAT and incoming PAT.

R1 ip route 192.168.176.0 255.255.255.0 192.168.168.236

R2 ip route 192.168.175.0 255.255.255.0 192.168.168.235

I found a really good example for multiple IPSEC tunnels that helped me. https://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/14133-ios-hub-spoke.pdf

I got the overload NAT and port mapping working. :-) Thanks for your help.

localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#sh run
Building configuration...

Current configuration : 3766 bytes
!
! Last configuration change at 17:22:42 UTC Tue Dec 18 2018 by cisco
! NVRAM config last updated at 18:12:13 UTC Tue Dec 18 2018 by cisco
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname localrtr
!
boot-start-marker
boot system flash:c2900-universalk9-mz.SPA.154-3.M1.bin
boot-end-marker
!
!
enable secret 5 $1$A3Kg$TZeqZI6QF3r.S4nu80fZJ1
!
no aaa new-model
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
no ip dhcp conflict logging
ip dhcp excluded-address 192.168.175.1 192.168.175.2
ip dhcp excluded-address 192.168.175.64 192.168.175.255
!
ip dhcp pool DHCPPOOL
network 192.168.175.0 255.255.255.0
dns-server 192.168.168.200 192.168.168.1
default-router 192.168.175.1
lease 0 2
!
!
!
no ip domain lookup
ip domain name cortana.com
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
cts logging verbose
!
!
voice-card 0
!
!
!
!
!
!
!
!
license udi pid CISCO2921/K9 sn FTX1714AKGS
hw-module pvdm 0/0
!
!
!
username cisco privilege 0 password 7 05190900355E41060D
!
redundancy
!
!
!
!
!
ip ftp username ahuffman
ip ftp password 7 091D1C180E040408134D241B1C051B090D
!
!
crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp key firewallcx address 192.168.168.236
!
!
crypto ipsec transform-set TS esp-3des esp-md5-hmac
mode tunnel
!
!
!
crypto map CMAP 10 ipsec-isakmp
set peer 192.168.168.236
set transform-set TS
match address VPN_TRAFFIC
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
service-module enable
!
interface GigabitEthernet0/0
description OUTSIDE
ip address 192.168.168.235 255.255.255.0
ip access-group REMOTE_OUTSIDE_IN_ACL in
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
crypto map CMAP
!
interface GigabitEthernet0/1
description INSIDE
ip address 192.168.175.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
!
interface GigabitEthernet0/2
no ip address
duplex auto
speed auto
!
interface FastEthernet0/0/0
description MGNT_10_10_10_15
switchport access vlan 200
no ip address
!
interface FastEthernet0/0/1
no ip address
!
interface FastEthernet0/0/2
no ip address
!
interface FastEthernet0/0/3
no ip address
!
interface Vlan1
no ip address
!
interface Vlan200
ip address 10.10.10.15 255.255.255.224
!
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip nat inside source static tcp 192.168.175.66 80 interface GigabitEthernet0/0 8080
ip nat inside source list 100 interface GigabitEthernet0/0 overload
ip route 0.0.0.0 0.0.0.0 192.168.168.1
ip route 192.168.176.0 255.255.255.0 192.168.168.236
!
ip access-list extended REMOTE_OUTSIDE_IN_ACL
permit tcp host 192.168.168.140 host 192.168.168.235 eq 8080
deny   tcp any host 192.168.168.235 eq 8080
deny   udp any host 192.168.168.235 eq 8080
permit ip any any
ip access-list extended VPN_TRAFFIC
permit ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255
!
!
!
access-list 100 remark -=[Define NAT Service]=-
access-list 100 deny   ip 192.168.175.0 0.0.0.255 192.168.176.0 0.0.0.255
access-list 100 permit ip 192.168.175.0 0.0.0.255 any
access-list 100 remark
!
control-plane
!
!
!
!
!
!
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default
!
!
!
!
!
!
!
gatekeeper
shutdown
!
!
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
login local
transport input ssh
!
scheduler allocate 20000 1000
ntp server 192.168.168.200
ntp server 10.10.10.30
!
end

localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#
localrtr#

Richard Burts · ‎12-18-2018

Allan

I am glad that you found a helpful reference for multiple tunnels. Thank you for sharing it with us. Hopefully some other participants in the community will also find it helpful.

This community can be a very helpful resource. And with the community you are not alone. I hope to see you continue to be active in the community.

HTH

Rick

HTH

Rick