ASA to ISR l2l VPN not working

[GeekTechnica]

Hi there,

I'm pretty new ASAs, and new to Cisco VPN in general.  I'm trying to set up a site to site tunnel.  The tunnel is up and traffic is passing one way ( I think) but not able to return.  

The Colo location has an ISR 2921 which has several tunnels to branch offices which seem to be working.  The ASA is in a new branch office.  The colo is on 192.168.170.0/24, the branch office is on 192.168.254.120/29 and connected to a L3 switch which is hosting the subnet 192.168.100.0/22

If I initiate a ping from the branch office switch to an IP on the colo side I get no reply but I do see the ACL on the ISR side increment:

BOSTON-SW01#ping 192.168.170.250 so vlan 100
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.170.250, timeout is 2 seconds:
Packet sent with a source address of 192.168.254.121 
.....
Success rate is 0 percent (0/5)
BOSTON-SW01#ping 192.168.170.250 so 192.168.100.21
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.170.250, timeout is 2 seconds:
Packet sent with a source address of 192.168.100.21 
.....
Success rate is 0 percent (0/5)

ISR:

COLO-ISR01#show access-lists 173
Extended IP access list 173
    10 permit ip any 172.20.43.0 0.0.0.255
    20 permit ip any 192.168.100.0 0.0.3.255 (4 matches)
    30 permit ip any 192.168.254.120 0.0.0.7 (54 matches)

 So it definitely looks like the traffic is making it across the tunnel.  Additionally a packet tracer on the ASA completes successfully:

BOSTON-ASA# packet-tracer input inside icmp 192.168.100.201 8 0 192.168.170.250

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 50.195.44.110 using egress ifc  outside

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,outside) source static local-network local-network destination static remote-network remote-network no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface outside
Untranslate 192.168.170.250/0 to 192.168.170.250/0

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any 
Additional Information:

Phase: 4
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (inside,outside) source static local-network local-network destination static remote-network remote-network no-proxy-arp route-lookup
Additional Information:
Static translate 192.168.100.201/0 to 192.168.100.201/0

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp 
service-policy global_policy global
Additional Information:

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (inside,outside) source static local-network local-network destination static remote-network remote-network no-proxy-arp route-lookup
Additional Information:

Phase: 11
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: NAT
Subtype: per-session
Result: ALLOW
Config:       
Additional Information:

Phase: 13
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 14
Type: FLOW-CREATION
Subtype: 
Result: ALLOW
Config:
Additional Information:
New flow created with id 21286, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Any help would be greatly appreciated.  I've just about run out of ideas.

Colo ISR Config:

version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname COLO-ISR01
!
boot-start-marker
boot-end-marker
!
!
logging buffered 52000
logging rate-limit all 10
enable secret xxxxxx
!
aaa new-model
!
!
aaa authentication login default local group radius
!
!
!
!
!
aaa session-id common
!
clock timezone EST -5 0
clock summer-time EDT recurring
!
no ipv6 cef
ip source-route
ip cef
!
!
!
no ip dhcp use vrf connected
!
!
ip flow-cache timeout active 1
ip domain name xxxxxx
ip name-server 192.168.170.243
!
multilink bundle-name authenticated
!
!
crypto pki token default removal timeout 0
!
crypto pki trustpoint TP-self-signed-4215859666
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-4215859666
 revocation-check none
 rsakeypair TP-self-signed-4215859666
!
!
crypto pki certificate chain TP-self-signed-4215859666
 certificate self-signed 01
  3082024D 308201B6 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 34323135 38353936 3636301E 170D3132 30313133 31343133
  31335A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D34 32313538
  35393636 3630819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
  8100C362 E90E6F88 EB3DADE1 039660D9 97A0D581 EFFF7908 2CFA6552 0D9A02E7
  2936AAF3 24298A6C E1F7A1B8 B2E4F38C 6DA5C920 2B557690 69FBD82A 6A6C06B0
  1FE8A0C8 CEE5787E 710BBEBB D42B97E2 2237EB4C 0E07B0D7 552CD417 CA1CA76C
  0539F989 40F1822C F549B836 C023E714 E5A64E40 24422C23 5B34AFF3 1FC4382D
  42C50203 010001A3 75307330 0F060355 1D130101 FF040530 030101FF 30200603
  551D1104 19301782 15434F4C 4F2E6E61 74696F6E 616C636F 72702E63 6F6D301F
  0603551D 23041830 168014DE 0A89F2A1 D6E038FC 603FF735 B46AD184 F1684E30
  1D060355 1D0E0416 0414DE0A 89F2A1D6 E038FC60 3FF735B4 6AD184F1 684E300D
  06092A86 4886F70D 01010405 00038181 00079CAC 098F2A73 38ADACDD 18D54B0C
  4EC51525 F5BBD170 2AE65685 C3EA5C3D F5BA2B37 2C192004 20A6327D 8B621932
  4BF1869D 39FFDA44 884A666E A4B413BE 999A2311 E37A639D CD390DF0 FD69A129
  3D87C08D B56BAE52 9F8894DB 0307E25C 4346BCE3 DC691709 03EF16D3 6333A20F
  A21CA1EB 9D7D4791 CE5BE4B2 B3BE4BB1 17
        quit
license udi pid CISCO2921/K9 sn FTX1624AK5P
!
!
object-group network NOG-RingCentral
 description All RingCentral Networks a/o 20170919
 103.44.68.0 255.255.252.0
 104.245.56.0 255.255.248.0
 185.23.248.0 255.255.252.0
 192.209.24.0 255.255.248.0
 199.255.120.0 255.255.252.0
 199.68.212.0 255.255.252.0
 208.87.40.0 255.255.252.0
!
object-group service SOG-RC-SIP
 description RingCentral SIP service identifiers a/o 20170919
 tcp-udp source range 5060 6000
 tcp-udp range 5060 6000
!
username xxxxx
!
redundancy
!
!
!
!
ip ssh version 2
!
track 1 interface GigabitEthernet0/0 line-protocol
!
class-map match-any CM-RTR-IB-RC-Other
 description AllRingCentral Originated Traffic
 match access-group name ACL-RTR-IB-RC-Networks-All
class-map match-any CM-RTR-IB-RC-SIP
 description RingCentral SIP Traffic
 match access-group name ACL-RTR-IB-RC-GeneralSIP
class-map match-any CM-GEN-OB-RC-Other
 description Elevated Priority
 match ip dscp af21
 match ip precedence 2
class-map match-any CM-GEN-OB-Video
 description Interactive Video
 match ip dscp af41
 match ip precedence 4
 match access-group name ACL-RoutingProtocol
class-map match-any CM-RTR-IB-RC-Video-RT
 description RingCentral Originated Traffic Video RTP
 match access-group name ACL-RTR-IB-RC-Video-RTP
class-map match-any CM-GEN-OB-Signaling
 description Call-Signaling
 match ip dscp af31
 match ip precedence 3
class-map match-any CM-RTR-IB-RC-Voice-RT
 description RingCentral Originated Traffic Voice RTP
 match access-group name ACL-RTR-IB-RC-Voice-RTP
class-map match-any CM-RTR-IB-Cust-AF12
 description Customer AF12 class traffic
 match access-group name ACL-RTR-IB-CustAF12
class-map match-any CM-GEN-OB-RT
 description Real-Time Traffic
 match ip dscp ef
 match ip precedence 5
class-map match-any CM-RTR-IB-Cust-AF13
 description Customer AF13 class traffic
 match access-group name ACL-RTR-IB-CustAF13
class-map match-any CM-RTR-IB-Cust-AF11
 description Customer AF11 class traffic
 match access-group name ACL-RTR-IB-CustAF11
class-map match-any CM-GEN-OB-Cust-AF11
 match ip dscp af11
class-map match-any CM-GEN-OB-Cust-AF12
 match ip dscp af12
class-map match-any CM-GEN-OB-Cust-AF13
 match ip dscp af13
!
!
policy-map PM-OB-RCFeed-QoS
 class CM-GEN-OB-RT
  set dscp ef
  priority percent 75
 class CM-GEN-OB-Video
  set dscp af41
  bandwidth percent 10
 class CM-GEN-OB-Signaling
  set dscp af31
  bandwidth percent 9
 class CM-GEN-OB-RC-Other
  set dscp af21
  bandwidth percent 5
 class class-default
  set dscp default
policy-map PM-RTR-OB-ToRC-100M
 class class-default
  service-policy PM-OB-RCFeed-QoS
policy-map PM-RTR-IB-Standard-QoS
 class CM-RTR-IB-RC-Voice-RT
  set dscp ef
 class CM-RTR-IB-RC-Video-RT
  set dscp af41
 class CM-RTR-IB-RC-SIP
  set dscp af31
 class CM-RTR-IB-RC-Other
  set dscp af21
 class CM-RTR-IB-Cust-AF13
  set dscp af13
 class CM-RTR-IB-Cust-AF12
  set dscp af12
 class CM-RTR-IB-Cust-AF11
  set dscp af11
 class class-default
  set dscp default
policy-map PM-GEN-OB-20-15-5-10
 class CM-GEN-OB-RT
  set dscp ef
  priority percent 20
 class CM-GEN-OB-Video
  set dscp af41
  bandwidth percent 15
 class CM-GEN-OB-Signaling
  set dscp af31
  bandwidth percent 5
 class CM-GEN-OB-RC-Other
  set dscp af21
  bandwidth percent 10
 class CM-GEN-OB-Cust-AF13
  set dscp af13
  bandwidth percent 5
 class CM-GEN-OB-Cust-AF12
  set dscp af12
  bandwidth percent 5
 class CM-GEN-OB-Cust-AF11
  set dscp af11
  bandwidth percent 5
 class class-default
  set dscp default
policy-map PM-RTR-OB-ToISP-100M
 class class-default
  shape average 95000000
  service-policy PM-GEN-OB-20-15-5-10
!
!
!
crypto isakmp policy 10
 encr 3des
 hash md5
 authentication pre-share
 group 2
crypto isakmp key xxxxx address 50.193.128.41
crypto isakmp key xxxxx address 69.193.221.154
crypto isakmp key xxxxx address 173.10.7.249
crypto isakmp key xxxxx address 4.59.148.125
crypto isakmp key xxxxx address 72.43.4.82
crypto isakmp key xxxxx address 50.252.159.221
crypto isakmp key xxxxx address 104.59.39.129
crypto isakmp key xxxxx address 97.105.97.234
crypto isakmp key xxxxx address 70.63.100.74
crypto isakmp key xxxxx address 96.92.217.81
crypto isakmp key xxxxx address 96.83.211.169
crypto isakmp key xxxxx address 96.67.206.169
crypto isakmp key xxxxx address 96.67.168.241
crypto isakmp key xxxxx address 96.79.18.193
crypto isakmp key xxxxx address 50.195.44.107
!
crypto ipsec security-association lifetime seconds 86400
!
crypto ipsec transform-set NCR-Transform esp-3des esp-md5-hmac
!
crypto map NCR-MAP 10 ipsec-isakmp
 description Tunnel to ISR-Newark
 set peer 96.83.211.169
 set transform-set NCR-Transform
 match address 155
crypto map NCR-MAP 20 ipsec-isakmp
 description Tunnel to ISR-Minneapolis
 set peer 96.67.168.241
 set transform-set NCR-Transform
 match address 156
crypto map NCR-MAP 30 ipsec-isakmp
 description Tunnel to ISR-Charlotte
 set peer 70.63.100.74
 set transform-set NCR-Transform
 match address 157
crypto map NCR-MAP 40 ipsec-isakmp
 description Tunnel to ISR-Chicago
 set peer 96.79.18.193
 set transform-set NCR-Transform
 match address 158
crypto map NCR-MAP 50 ipsec-isakmp
 description Tunnel to ISR-Springfield
 set peer 96.92.217.81
 set transform-set NCR-Transform
 match address 159
crypto map NCR-MAP 60 ipsec-isakmp
 description Tunnel to ISR-WashDC
 set peer 4.59.148.125
 set transform-set NCR-Transform
 match address 160
crypto map NCR-MAP 70 ipsec-isakmp
 description Tunnel to ISR-Dover
 set peer 173.10.7.249
 set transform-set NCR-Transform
 match address 161
crypto map NCR-MAP 80 ipsec-isakmp
 description Tunnel to ISR-NyCity
 set peer 69.193.221.154
 set transform-set NCR-Transform
 match address 162
crypto map NCR-MAP 90 ipsec-isakmp
 description Tunnel to ISR-LosAngeles
 set peer 104.59.39.129
 set transform-set NCR-Transform
 match address 163
crypto map NCR-MAP 100 ipsec-isakmp
 description Tunnel to ISR-Sacramento
 set peer 96.67.206.169
 set transform-set NCR-Transform
 match address 164
crypto map NCR-MAP 110 ipsec-isakmp
 description Tunnel to ISR-Dungeon
 set peer 50.193.128.41
 set transform-set NCR-Transform
 match address 165
crypto map NCR-MAP 120 ipsec-isakmp
 description Tunnel to ISR-Albany
 set peer 72.43.4.82
 set transform-set NCR-Transform
 match address 166
crypto map NCR-MAP 150 ipsec-isakmp
 description Tunnel to ISR-Tallahassee
 set peer 50.252.159.221
 set transform-set NCR-Transform
 match address 169
crypto map NCR-MAP 170 ipsec-isakmp
 description Tunnel to Dallas-VPN
 set peer 97.105.97.234
 set transform-set NCR-Transform
 match address 171
crypto map NCR-MAP 180 ipsec-isakmp
 description Tunnel to Boston-ASA
 set peer 50.195.44.107
 set transform-set NCR-Transform
 match address 173
!
!
!
!
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
interface GigabitEthernet0/0
 description Primary Circuit TWTC Circuit 01/KDGS/001983/TWCS
 bandwidth 75000
 ip address 64.129.18.30 255.255.255.252
 ip flow ingress
 ip flow egress
 duplex auto
 speed auto
 service-policy input PM-RTR-IB-Standard-QoS
 service-policy output PM-RTR-OB-ToISP-100M
!
interface GigabitEthernet0/1
 description SW-01:Gi0/44
 no ip address
 duplex full
 speed 1000
 service-policy output PM-GEN-OB-20-15-5-10
!
interface GigabitEthernet0/1.10
 description NATIVE/PC
 encapsulation dot1Q 10
 ip address 192.168.170.28 255.255.255.0
 ip flow ingress
 ip flow egress
 standby 1 ip 192.168.170.21
 standby 1 priority 101
 standby 1 preempt
 standby 1 track 1 decrement 5
!
interface GigabitEthernet0/1.20
 description VOICE
 encapsulation dot1Q 20
 ip address 192.168.171.24 255.255.255.0
 standby 2 ip 192.168.171.21
 standby 2 priority 101
 standby 2 preempt
!
interface GigabitEthernet0/1.30
 description COLO-OFFICE
 encapsulation dot1Q 30
 ip address 192.168.169.24 255.255.255.0
 standby 1 preempt
 standby 3 ip 192.168.169.21
 standby 3 priority 101
!
interface GigabitEthernet0/1.40
 encapsulation dot1Q 40
 shutdown
!
interface GigabitEthernet0/2
 description Secondary Circuit TWTC 01/KEFN/103161/TWCS
 ip address 173.226.250.250 255.255.255.248
 ip access-group NCR_OFFICES in
 ip nat outside
 ip virtual-reassembly in
 duplex full
 speed 100
 crypto map NCR-MAP
!
router bgp 64539
 bgp log-neighbor-changes
 network 172.20.43.0 mask 255.255.255.0
 network 192.168.168.0 mask 255.255.252.0
 redistribute static route-map Set-Metric
 neighbor 64.129.18.29 remote-as 4323
 neighbor 64.129.18.29 prefix-list Outbound-Filter out
 neighbor 64.129.18.29 route-map Set-Metric out
 default-information originate
!
no ip forward-protocol nd
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip flow-export source GigabitEthernet0/1.10
ip flow-export version 5
ip flow-export destination 192.168.170.250 2055
!
ip route 0.0.0.0 0.0.0.0 192.168.170.22
ip route 4.30.32.182 255.255.255.255 173.226.250.249
ip route 4.30.58.82 255.255.255.255 173.226.250.249
ip route 4.59.148.125 255.255.255.255 173.226.250.249
ip route 50.193.128.41 255.255.255.255 173.226.250.249
ip route 50.195.44.107 255.255.255.255 173.226.250.249
ip route 50.252.159.216 255.255.255.248 173.226.250.249
ip route 69.193.221.154 255.255.255.255 173.226.250.249
ip route 70.63.100.74 255.255.255.255 173.226.250.249
ip route 72.43.4.82 255.255.255.255 173.226.250.249
ip route 96.67.168.240 255.255.255.240 173.226.250.249
ip route 96.67.206.168 255.255.255.248 173.226.250.249
ip route 96.79.18.192 255.255.255.248 173.226.250.249
ip route 96.83.211.169 255.255.255.255 173.226.250.249
ip route 96.92.217.81 255.255.255.255 173.226.250.249
ip route 97.105.97.232 255.255.255.248 173.226.250.249
ip route 104.59.39.128 255.255.255.248 173.226.250.249
ip route 172.20.43.0 255.255.255.0 192.168.170.22
ip route 173.10.7.249 255.255.255.255 173.226.250.249
ip route 173.10.139.98 255.255.255.255 173.226.250.249
ip route 192.168.4.0 255.255.252.0 173.226.250.249 150
ip route 192.168.8.0 255.255.252.0 173.226.250.249 150
ip route 192.168.20.0 255.255.252.0 173.226.250.249 150
ip route 192.168.28.0 255.255.252.0 173.226.250.249 150
ip route 192.168.40.0 255.255.252.0 173.226.250.249 150
ip route 192.168.48.0 255.255.252.0 173.226.250.249 150
ip route 192.168.60.0 255.255.252.0 173.226.250.249 150
ip route 192.168.80.0 255.255.252.0 173.226.250.249 150
ip route 192.168.120.0 255.255.252.0 173.226.250.249 150
ip route 192.168.132.0 255.255.252.0 173.226.250.249 150
ip route 192.168.140.0 255.255.252.0 173.226.250.249 150
ip route 192.168.148.0 255.255.252.0 173.226.250.249 150
ip route 192.168.152.0 255.255.252.0 173.226.250.249 150
ip route 192.168.160.0 255.255.252.0 173.226.250.249 150
ip route 192.168.168.0 255.255.252.0 Null0
ip route 192.168.180.0 255.255.252.0 173.226.250.249 150
ip route 192.168.184.0 255.255.255.0 192.168.170.26
ip route 192.168.254.16 255.255.255.248 173.226.250.249 150
ip route 192.168.254.24 255.255.255.248 173.226.250.249 150
ip route 192.168.254.32 255.255.255.248 173.226.250.249 150
ip route 192.168.254.40 255.255.255.248 173.226.250.249 150
ip route 192.168.254.48 255.255.255.248 173.226.250.249 150
ip route 192.168.254.56 255.255.255.248 173.226.250.249 150
ip route 192.168.254.64 255.255.255.248 173.226.250.249 150
ip route 192.168.254.72 255.255.255.248 173.226.250.249 150
ip route 192.168.254.80 255.255.255.248 173.226.250.249 150
ip route 192.168.254.88 255.255.255.248 173.226.250.249 150
ip route 192.168.254.96 255.255.255.248 173.226.250.249 150
ip route 192.168.254.104 255.255.255.248 173.226.250.249 150
ip route 192.168.254.112 255.255.255.248 173.226.250.249 150
ip route 210.5.169.133 255.255.255.255 173.226.250.249
!
ip access-list extended ACL-RTR-IB-Cust-AF11
 remark Identify customer traffic for AF11 Classification
 deny   udp any any
 deny   tcp any any
ip access-list extended ACL-RTR-IB-RC-GeneralSIP
 remark RingCentral SIP Signaling a/o 20170919
 permit object-group SOG-RC-SIP object-group NOG-RingCentral any
ip access-list extended ACL-RTR-IB-RC-Networks-All
 remark RingCentral ALL traffic a/o 20170919
 permit ip object-group NOG-RingCentral any
 remark Identify Customer Traffic for AF11 Classification
ip access-list extended ACL-RTR-IB-RC-Video-RTP
 remark RingCentral Video Real-Time a/o 20170919
 permit udp object-group NOG-RingCentral any range 8801 8802
ip access-list extended ACL-RTR-IB-RC-Voice-RTP
 remark RingCentral Voice Real-Time a/o 20170919
 permit udp object-group NOG-RingCentral range 9000 64999 any
ip access-list extended ACL-RoutingProtocol
 permit udp any any eq rip
 permit udp any eq rip any
 permit eigrp any any
 permit ospf any any
 permit tcp any any eq bgp
 permit tcp any eq bgp any
ip access-list extended NCR_OFFICES
 permit icmp host 173.226.250.249 host 173.226.250.250
 remark Dover
 permit ip host 173.10.7.249 host 173.226.250.250
 remark Dungeon
 permit ip host 50.193.128.41 host 173.226.250.250
 remark NYCity
 permit ip host 69.193.221.154 host 173.226.250.250
 remark WashDC
 permit ip host 4.59.148.125 host 173.226.250.250
 remark Albany
 permit ip host 72.43.4.82 host 173.226.250.250
 remark Tallahassee
 permit ip host 50.252.159.221 host 173.226.250.250
 remark Chicago
 permit ip host 96.79.18.193 host 173.226.250.250
 remark Los Angeles
 permit ip host 104.59.39.129 host 173.226.250.250
 remark Dallas
 permit ip host 97.105.97.234 host 173.226.250.250
 remark Charlotte
 permit ip host 70.63.100.74 host 173.226.250.250
 remark Springfield
 permit ip host 96.92.217.81 host 173.226.250.250
 remark Minneapolis
 permit ip host 96.67.168.241 host 173.226.250.250
 remark Newark
 permit ip host 96.83.211.169 host 173.226.250.250
 remark Sacramento
 permit ip host 96.67.206.169 host 173.226.250.250
 remark Boston
 permit ip host 50.195.44.107 host 173.226.250.250
 deny   ip any any
!
!
ip prefix-list Outbound-Filter seq 5 permit 192.168.168.0/22
ip prefix-list Outbound-Filter seq 10 permit 0.0.0.0/0
ip prefix-list Outbound-Filter seq 15 permit 192.168.0.0/16
!
ip prefix-list Set-Metric seq 10 permit 192.168.0.0/16
ip prefix-list Set-Metric seq 20 permit 0.0.0.0/0
ip prefix-list Set-Metric seq 30 permit 192.168.168.0/22
no logging trap
access-list 23 remark : VTY control
access-list 23 remark : ASA interface
access-list 23 permit 64.128.232.98
access-list 23 remark : Dungeon external circuit
access-list 23 permit 50.193.128.41
access-list 23 remark : NCR internal Admin VPN
access-list 23 permit 172.30.16.0 0.0.0.255
access-list 23 remark : NCR internal
access-list 23 permit 192.168.0.0 0.0.255.255
access-list 23 deny   any log
access-list 133 remark : trap bad IPs
access-list 133 deny   ip 169.254.0.0 0.0.255.255 any
access-list 133 permit ip any any
access-list 155 remark : Newark
access-list 155 permit ip any 192.168.140.0 0.0.3.255
access-list 155 permit ip any 192.168.254.72 0.0.0.7
access-list 156 remark : Minneapolis
access-list 156 permit ip any 192.168.148.0 0.0.3.255
access-list 156 permit ip any 192.168.254.32 0.0.0.7
access-list 156 permit ip any 192.168.152.0 0.0.3.255
access-list 157 remark : Charlotte
access-list 157 permit ip any 192.168.180.0 0.0.3.255
access-list 157 permit ip any 192.168.254.8 0.0.0.7
access-list 158 remark : Chicago
access-list 158 permit ip any 192.168.120.0 0.0.3.255
access-list 158 permit ip any 192.168.254.104 0.0.0.7
access-list 159 remark : Springfield
access-list 159 permit ip any 192.168.160.0 0.0.3.255
access-list 159 permit ip any 192.168.254.64 0.0.0.7
access-list 160 remark : WashDC
access-list 160 permit ip any 192.168.80.0 0.0.3.255
access-list 160 permit ip any 192.168.254.56 0.0.0.7
access-list 161 remark : Dover
access-list 161 permit ip any 192.168.8.0 0.0.3.255
access-list 161 permit ip any 192.168.254.48 0.0.0.7
access-list 162 remark : NyCity
access-list 162 permit ip any 192.168.4.0 0.0.3.255
access-list 162 permit ip any 192.168.254.24 0.0.0.7
access-list 162 permit ip 172.20.43.0 0.0.0.255 192.168.4.0 0.0.0.255
access-list 163 remark : LosAngeles
access-list 163 permit ip any 192.168.40.0 0.0.3.255
access-list 163 permit ip any 192.168.254.88 0.0.0.7
access-list 164 remark : Sacramento
access-list 164 permit ip any 192.168.48.0 0.0.3.255
access-list 164 permit ip any 192.168.254.40 0.0.0.7
access-list 165 remark : Dungeon
access-list 165 permit ip any 192.168.28.0 0.0.3.255
access-list 165 permit ip any 192.168.254.80 0.0.0.7
access-list 165 permit ip 172.20.43.0 0.0.0.255 192.168.30.0 0.0.0.255
access-list 166 remark : Albany
access-list 166 permit ip any 192.168.20.0 0.0.3.255
access-list 166 permit ip any 192.168.254.16 0.0.0.7
access-list 169 remark : Tallahassee
access-list 169 permit ip any 192.168.60.0 0.0.3.255
access-list 169 permit ip any 192.168.254.96 0.0.0.7
access-list 171 remark : Dallas-VPN
access-list 171 permit ip any 192.168.132.0 0.0.3.255
access-list 171 permit ip any 192.168.254.112 0.0.0.7
access-list 173 remark : Boston-VPN
access-list 173 permit ip any 172.20.43.0 0.0.0.255
access-list 173 permit ip any 192.168.100.0 0.0.3.255
access-list 173 permit ip any 192.168.254.120 0.0.0.7
access-list 199 remark TEMP-PBR-ACL-DOV
access-list 199 permit ip host 192.168.170.190 host 192.168.10.52
access-list 199 permit ip host 192.168.170.225 host 192.168.10.52
!
!
!
!
route-map TEMP-PBR-RM-DOV permit 10
 match ip address 199
 set ip next-hop 66.195.78.17
!
route-map Set-Metric permit 10
 match ip address prefix-list Set-Metric
 set metric 100
!
!
snmp-server community xxxx RW
snmp-server ifindex persist
snmp-server location CoLo
radius-server host 192.168.170.203 key 7 xxxxx
!
!
!
control-plane
!
!
privilege exec level 15 connect
privilege exec level 15 telnet
privilege exec level 15 rlogin
privilege exec level 15 show ip access-lists
privilege exec level 1 show ip
privilege exec level 15 show access-lists
privilege exec level 15 show logging
privilege exec level 1 show
banner exec ^CC

NOTICE!!!     NOTICE!!!     NOTICE!!!     NOTICE!!!     NOTICE!!!

This system is solely for the use of authorized users performing
official duties. You have no expectation of privacy in its use
and to ensure that the system is functioning properly individuals
using this system are subject to having all of their activities
monitored and recorded. Use of this system evidences an express
consent to such monitoring and agreement that if such monitoring
reveals evidence of possible abuse or criminal activity the results
of such monitoring may be provided to the appropriate officials.

^C
banner login ^CC

WARNING!!!    WARNING!!!    WARNING!!!    WARNING!!!    WARNING!!!

This system is solely for the use of authorized users performing
official duties. You have no expectation of privacy in its use
and to ensure that the system is functioning properly individuals
using this system are subject to having all of their activities
monitored and recorded. Use of this system evidences an express
consent to such monitoring and agreement that if such monitoring
reveals evidence of possible abuse or criminal activity the results
of such monitoring may be provided to the appropriate officials.

^C
!
line con 0
 logging synchronous
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport input all
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 access-class 23 in
 exec-timeout 30 0
 privilege level 15
 logging synchronous
 transport input ssh
!
scheduler allocate 20000 1000
ntp server 192.168.170.243
end

ASA Config:

ASA Version 9.8(2) 
!
hostname BOSTON-ASA
enable password xxxxx
xlate per-session deny tcp any4 any4
xlate per-session deny tcp any4 any6
xlate per-session deny tcp any6 any4
xlate per-session deny tcp any6 any6
xlate per-session deny udp any4 any4 eq domain
xlate per-session deny udp any4 any6 eq domain
xlate per-session deny udp any6 any4 eq domain
xlate per-session deny udp any6 any6 eq domain
names

!
interface GigabitEthernet1/1
 description Comcast
 nameif outside
 security-level 0
 ip address 50.195.44.107 255.255.255.248 
!
interface GigabitEthernet1/2
 description MPLS
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/3
 nameif inside
 security-level 100
 ip address 192.168.254.123 255.255.255.248 
!
interface GigabitEthernet1/4
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/5
 shutdown
 no nameif    
 no security-level
 no ip address
!
interface GigabitEthernet1/6
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/7
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet1/8
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Management1/1
 management-only
 no nameif    
 no security-level
 no ip address
!
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
object-group network local-network
 network-object 192.168.100.0 255.255.252.0
 network-object 192.168.254.120 255.255.255.248
object-group network remote-network
 network-object 192.168.170.0 255.255.255.0
 network-object 172.20.43.0 255.255.255.0
access-list outside_access_in extended deny udp any4 any4 eq netbios-ns 
access-list outside_access_in extended deny udp any4 any4 eq netbios-dgm 
access-list outside_access_in extended deny tcp any4 any4 eq netbios-ssn 
access-list outside_access_in extended deny tcp any4 any4 eq 445 
access-list outside_access_in extended permit icmp any4 any4 log disable 
access-list inside_access_in extended permit ospf any4 any4 
access-list inside_access_in extended permit ip any any 
access-list asa-router-vpn extended permit ip object-group local-network object-group remote-network 
access-list inside_nat0_outobund extended permit ip object-group local-network object-group remote-network 
pager lines 24
mtu outside 1500
mtu inside 1500
icmp unreachable rate-limit 1 burst-size 1
icmp permit any echo-reply outside
icmp permit any outside
icmp permit any inside
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
arp rate-limit 16384
nat (inside,outside) source static local-network local-network destination static remote-network remote-network no-proxy-arp route-lookup
!
nat (inside,outside) after-auto source dynamic any interface
access-group outside_access_in in interface outside
access-group inside_access_in in interface inside
!
route outside 0.0.0.0 0.0.0.0 50.195.44.110 1
route inside 192.168.100.0 255.255.252.0 192.168.254.121 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
timeout conn-holddown 0:00:15
timeout igp stale-route 0:01:10
user-identity default-domain LOCAL
aaa authentication ssh console LOCAL 
aaa authentication login-history
http server enable
http 192.168.1.0 255.255.255.0 inside
http 64.128.232.96 255.255.255.224 outside
http 192.168.0.0 255.255.0.0 inside
no snmp-server location
no snmp-server contact
service sw-reset-button
crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac 
crypto ipsec security-association pmtu-aging infinite
crypto map outside_map 10 match address asa-router-vpn
crypto map outside_map 10 set peer 173.226.250.250 
crypto map outside_map 10 set ikev1 transform-set ESP-3DES-MD5
crypto map outside_map interface outside
crypto ca trustpool policy
crypto ikev1 enable outside
crypto ikev1 policy 10
 authentication pre-share
 encryption 3des
 hash md5
 group 2
 lifetime 86400
telnet timeout 5
ssh stricthostkeycheck
ssh 64.128.232.96 255.255.255.224 outside
ssh 192.168.0.0 255.255.0.0 inside
ssh timeout 5
ssh version 2
ssh key-exchange group dh-group1-sha1
console timeout 0

threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
group-policy DfltGrpPolicy attributes
 vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-clientless
dynamic-access-policy-record DfltAccessPolicy
username xxxxx
tunnel-group 173.226.250.250 type ipsec-l2l
tunnel-group 173.226.250.250 ipsec-attributes
 ikev1 pre-shared-key xxxxx
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum client auto
  message-length maximum 512
  no tcp-inspection
policy-map global_policy
 class inspection_default
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect ip-options 
  inspect netbios 
  inspect rsh 
  inspect rtsp 
  inspect skinny  
  inspect esmtp 
  inspect sqlnet 
  inspect sunrpc 
  inspect tftp 
  inspect sip  
  inspect xdmcp 
  inspect dns preset_dns_map 
  inspect icmp 
policy-map type inspect dns migrated_dns_map_2
 parameters
  message-length maximum client auto
  message-length maximum 512
  no tcp-inspection
policy-map type inspect dns migrated_dns_map_1
 parameters
  message-length maximum client auto
  message-length maximum 512
  no tcp-inspection
!
service-policy global_policy global
prompt hostname context 
no call-home reporting anonymous
call-home
 profile CiscoTAC-1
  no active   
  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
  destination address email callhome@cisco.com
  destination transport-method http
  subscribe-to-alert-group diagnostic
  subscribe-to-alert-group environment
  subscribe-to-alert-group inventory periodic monthly
  subscribe-to-alert-group configuration periodic monthly
  subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:91bc890529fddd925570c35b4baa1d1a
: end

[Sheraz.Salim]

Hi GreekTechnica,

From the ASA packet tracer output your configuration looks okay as its show packet is passing. in that case you can rule out if ASA is the issue. it would be great if you could provide a logical diagram of your network it would help me to understand and pin point the issue with your network. as you have put over information. which is good but also very easy for us to destruct. you need to proved the public ip address (fake the ip addresses) from the ISR to ASA.

to my understanding

BOSTON-SW01 is connected to BOSTON- ASA.

if this is correct, and you also did a packet-tracer on ASA to remote ip address, which ASA confirm it is allowed. here i would strongly suggest you to check if the switch had an route 192.168.170.0/24 as its next hope to ASA.

BOSTON-SW01

!
ip route 192.168.170.0 255.255.255.0 192.168.254.123

!

if the switch is a layer 2 than could you check your layer3 switch has a route.

when you ping from BOSTON-SW01 could you issue the command on ISR

show crypto isakmp sa peer x.x.x.x   (your ASA public IP) what output it show you?

[Sheraz.Salim]

From your ASA your configuration look good. as packet tracer confirm the traffic is allowed and the process of VPN encryption is happening. 

it would be great if you could provide a logical diagram of your network with Public ip address (Fake the Public IP). as you have put too much information but is very easy for us to get us lost. for example you have not mention what is the public ip address at your ISR pointing to ASA. just curious as per my understanding in between ASA there is a BOSTON-SW and you pinging from BOSTON SW to your remote network. if this is correct can you confirm the routing is placed accordingly. for example does your switch have a static route to your ASA.

BOSTON-SW

!

ip route 192.168.170.0 255.255.255.0 192.168.254.123

!

or if there is a layer 3 switch make sure you have a static route. it could be ASA know how to encrypt an decrypt (vpn configuration are correct) but he do not see the packet coming from inside going to remote side.

could you show these command while you in BOSTON-SW ping to remote network.

on ISR show crypto ipsec peer X.X.X.X     X=ASA public ip address.

[Mohammad Alhyari]

Hi, 

Since the traffic flow you are chasing is from the branch office where the ASA is located to the Colo side where the router is installed and the packet tracer looks fine on the ASA i suggest you proceed with the following for now:

1- share the output of show cry ipsec sa from both the ASA and the router to ensure that we have ESP packets leaving the ASA and received at the router. We should see encaps on the ASA and decaps on the router.

2-  after that and if we see encaps and decaps we will try to ping the router interface itself. From the devices behind the ASA try to ping the Router inside address and see if this one works.

3- we should have a route on the router that sends back to the outside interface when the destination is the inside interface of the ASA.

4- if the ping to the router works but to the inside network it fails then we need to check the network on the inside of the router, does it have a correct route back to the router when trying to reach the subnet behind the ASA.  

5- depending on what you find on point number 4 we may troubleshoot further data path issues on the router.

HTH.

Moh,