Debug ICMP Output

johnlloyd_13 · ‎01-22-2016

hi all,

i'm trying to debug some routing on an IPsec router. GRE over IPsec VPN works and establish but pings seem to fail and not sure if upstream client DSL router is blocking it. i already include a deny ACE for the VPN private IP for the NAT ACL on the 2911 remote router.

brief diagram:

GRE tunnel 701 - my 2911 IPsec router <> client DSL router (with NAT) <> internet <> my ASA FW <> my 3925 router - GRE tunnel 701

2911 REMOTE SITE

ip nat inside source list 150 interface GigabitEthernet0/2 overload

Extended IP access list 150   <<< NAT ACL
    10 deny ip host 13.161.4.151 host 13.164.30.7 (20 matches)
    20 deny ip 172.27.0.0 0.0.255.255 any
    30 permit ip 192.168.5.0 0.0.0.255 any
    40 permit ip 192.168.55.0 0.0.0.255 any

Extended IP access list OFC_TO_HUB <<< CRYPTO ACL
10 permit ip host 13.161.4.151 host 13.164.30.7 (657 matches)

interface Tunnel701
ip address 172.27.11.10 255.255.255.252
tunnel source 192.168.1.2
tunnel destination 13.164.30.7 <<< OUR 3925 ROUTER LOOPBACK IP

interface GigabitEthernet0/2
description Internet <<< CLIENT OWNED
ip address 192.168.1.2 255.255.255.0
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
crypto map CMAP

HUB 3925

interface Tunnel701
ip address 172.27.11.9 255.255.255.252
tunnel source 13.164.30.7
tunnel destination 13.161.4.151 << CLIENT DSL PUBLIC OUTSIDE IP; OUR 192.168.1.2 IS NAT'd/PORT FORWARD
end

also when i initiate my pings from my 3925 GRE source, the IPsec SA establishes but not the other way around. i can also see encrypt/decrypt packets to be asymmetric but my crypto ACL is already mirrored (besides IKE phase 1 and phase 2 is establised).

i can also see hits on my crypto ACL.

2911#sh crypto session
Crypto session current status

Interface: GigabitEthernet0/2
Session status: UP-ACTIVE
Peer: 13.164.30.6 port 4500
IKE SA: local 192.168.1.2/4500 remote 13.164.30.6/4500 Active
IPSEC FLOW: permit ip host 13.161.4.151 host 13.164.30.7
Active SAs: 2, origin: crypto map

2911# sh crypto isa sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
192.168.1.2 13.164.30.6 QM_IDLE 1046 ACTIVE <<< OUR ASA OUTSIDE IP IS 13.164.30.6

2911#sh crypto ipsec sa

interface: GigabitEthernet0/2
    Crypto map tag: RNCMAP, local addr 192.168.1.2

   protected vrf: (none)
   local ident (addr/mask/prot/port): (13.161.4.151/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (13.164.30.7/255.255.255.255/0/0)
   current_peer 13.164.30.6 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 12, #pkts decrypt: 12, #pkts verify: 12

2911 debug

# ping 172.27.11.9 so 172.27.11.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.27.11.9, timeout is 2 seconds:
Packet sent with a source address of 172.27.11.10

*Jan 23 00:57:07.999: IP: s=172.27.11.10 (local), d=172.27.11.9, len 100, local feature
*Jan 23 00:57:07.999:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Jan 23 00:57:07.999: FIBipv4-packet-proc: route packet from (local) src 172.27.11.10 dst 172.27.11.9
*Jan 23 00:57:07.999: FIBipv4-packet-proc: packet routing succeeded
*Jan 23 00:57:07.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending
*Jan 23 00:57:07.999:     ICMP type=8, code=0
*Jan 23 00:57:07.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending full packet
*Jan 23 00:57:07.999:     ICMP type=8, code=0.
*Jan 23 00:57:09.999: IP: s=172.27.11.10 (local), d=172.27.11.9, len 100, local feature
*Jan 23 00:57:09.999:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE <<< CAN SOMEONE ADVISE WHAT ARE THESE OUTPUT?
*Jan 23 00:57:09.999: FIBipv4-packet-proc: route packet from (local) src 172.27.11.10 dst 172.27.11.9
*Jan 23 00:57:09.999: FIBipv4-packet-proc: packet routing succeeded    <<< DOES THIS MEAN MY ROUTING IS FINE AND CLIENT ROUTER BLOCKS ICMP?
*Jan 23 00:57:09.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending
*Jan 23 00:57:09.999:     ICMP type=8, code=0
*Jan 23 00:57:09.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending full packet
*Jan 23 00:57:09.999:     ICMP type=8, code=0.
*Jan 23 00:57:11.999: IP: s=172.27.11.10 (local), d=172.27.11.9, len 100, local feature
*Jan 23 00:57:11.999:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Jan 23 00:57:11.999: FIBipv4-packet-proc: route packet from (local) src 172.27.11.10 dst 172.27.11.9
*Jan 23 00:57:11.999: FIBipv4-packet-proc: packet routing succeeded
*Jan 23 00:57:11.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending
*Jan 23 00:57:11.999:     ICMP type=8, code=0
*Jan 23 00:57:11.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending full packet
*Jan 23 00:57:11.999:     ICMP type=8, code=0.
*Jan 23 00:57:13.999: IP: s=172.27.11.10 (local), d=172.27.11.9, len 100, local feature
*Jan 23 00:57:13.999:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Jan 23 00:57:13.999: FIBipv4-packet-proc: route packet from (local) src 172.27.11.10 dst 172.27.11.9
*Jan 23 00:57:13.999: FIBipv4-packet-proc: packet routing succeeded
*Jan 23 00:57:13.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending
*Jan 23 00:57:13.999:     ICMP type=8, code=0
*Jan 23 00:57:13.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending full packet
*Jan 23 00:57:13.999:     ICMP type=8, code=0.
*Jan 23 00:57:15.999: IP: s=172.27.11.10 (local), d=172.27.11.9, len 100, local feature
*Jan 23 00:57:15.999:     ICMP type=8, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Jan 23 00:57:15.999: FIBipv4-packet-proc: route packet from (local) src 172.27.11.10 dst 172.27.11.9
*Jan 23 00:57:15.999: FIBipv4-packet-proc: packet routing succeeded
*Jan 23 00:57:15.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending
*Jan 23 00:57:15.999:     ICMP type=8, code=0
*Jan 23 00:57:15.999: IP: s=172.27.11.10 (local), d=172.27.11.9 (Tunnel701), len 100, sending full packet
*Jan 23 00:57:15.999:     ICMP type=8, code=0.
Success rate is 0 percent (0/5)

----

HUB ASA

asa/admin# sh crypto isa sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 13.161.4.151
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE

asa/admin# sh crypto ipsec sa
interface: outside
    Crypto map tag: VPN_CMAP, seq num: 701, local addr: 13.164.30.6

      access-list HUB-PE01_2911 extended permit ip host 13.164.30.6 host 13.161.4.151
      local ident (addr/mask/prot/port): (13.164.30.7/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (13.161.4.151 /255.255.255.255/0/0)
      current_peer: 13.161.4.151

      #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
      #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 4, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

Philip D'Ath · ‎01-22-2016

The Tunnel source and destination must match the crypto map, and the real interface IP addresses as a result. The NAT exclusion must also use these same IP addresses (the actual real IP addresses configured on the interfaces).

So make the tunnel destination on the hub 192.168.1.2. The encryption domain for both devices must be 192.168.1.2/32 and 13.164.30.7/32.

The hub must be set to route 192.168.1.2 via the Internet next hop (if it isn't already).

On the hub make the "set peer" in the crypto map point to the public IP address of the DSL router. The isakmp key address should also use the public IP address of the DSL router on the hub.

If you can't get it, post a more fuller config.

johnlloyd_13 · ‎01-23-2016

hi,

the 192.168.1.2 is client's private IP. we don't use 192.168.1.x on hub router/network.

i wouldn't be able to establish an IPsec SA (IKE phase 1 and 2) if i didn't have the 'mirrored' cyrpto ACL/encryption domain.

i think i've tried using 192.168.1.2/32 and 13.164.30.7/32 for the encryption domain/crypto ACL. but i'll try again and post result.

edit:

i tried suggested encryption domain and it still ping can't ping the remote GRE tunnel.

though i initiate IPsec on 2911 now (before can only initiate on hub 3925) and could see some ACL matches on the deny NAT IP subnets.

the encrypt/decrypt is still asymmetric.

when i removed the crypto map on client facing DSL router G0/2, i can ping the remote side but only when using the client's source IP 192.168.1.2, so maybe plain GRE works?

2911#sh crypto sess
Crypto session current status

Interface: GigabitEthernet0/2
Session status: UP-ACTIVE
Peer: 13.164.30.6 port 4500
IKE SA: local 192.168.1.2/4500 remote 13.164.30.6/4500 Active
IPSEC FLOW: permit ip host 192.168.1.2 host 13.164.30.7
        Active SAs: 2, origin: crypto map

2911#sh crypto isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
13.164.30.6 192.168.1.2     QM_IDLE           1053 ACTIVE

interface: GigabitEthernet0/2
    Crypto map tag: RNCMAP, local addr 192.168.1.2

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.1.2/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (13.164.30.7/255.255.255.255/0/0)
   current_peer 13.164.30.6 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0

2911#sh run int tu701
Building configuration...

Current configuration : 172 bytes
!
interface Tunnel701
ip address 172.27.11.10 255.255.255.252
tunnel source 192.168.1.2
tunnel destination 13.164.30.7
!
end

2911#ping 13.164.30.7 so 192.168.1.2   <<< CMPA ON G0/2 OUTSIDE INT REMOVED

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 13.164.30.7, timeout is 2 seconds:
Packet sent with a source address of 192.168.1.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/8 ms
rosneft-hcm-gw01#ping 172.27.11.9

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.27.11.9, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Extended IP access list 150
    10 deny ip host 192.168.1.2 host 13.164.30.7 (4 matches)
    20 deny ip 172.27.0.0 0.0.255.255 any (2 matches)
    30 permit ip 192.168.5.0 0.0.0.255 any
    40 permit ip 192.168.55.0 0.0.0.255 any

Extended IP access list OFC_TO_HUB
    10 permit ip host 192.168.1.2 host 13.164.30.6 (9 matches)

----

HUB

asa# sh crypto isa sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 13.161.4.151
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE

asa# sh crypto ipsec sa
interface: outside
    Crypto map tag: VPN_CMAP, seq num: 701, local addr: 13.164.30.6

      access-list VTC1-PE01_ROSNEFT-GW01 extended permit ip host 13.164.30.7 host 192.168.1.2
      local ident (addr/mask/prot/port): (13.164.30.7/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (192.168.1.2/255.255.255.255/0/0)
      current_peer: 13.161.4.151

      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
      #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
      #pkts compressed: 0, #pkts decompressed: 0

3925

interface Tunnel701
ip address 172.27.11.9 255.255.255.252
tunnel source 13.164.30.7
tunnel destination 192.168.1.2

3925#ping 172.27.11.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.27.11.10, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Philip D'Ath · ‎01-23-2016

I'm confused around the ASA and the 3925. Is the ASA or the 3925 terminating the IPSec?

If the ASA is terminating the IPSec, then what is the outside IP address of the 3925, because it looks like 13.164.30.7 is on the ASA.

Your output below shows the GRE over IPSec packets are making it from the remote site to the Hub, but not from the Hub to the remote site.

If the ASA is terminating the IPSec, then:

The Hub tunnel source must be an interface on the 3925.
The ASA must have a route to this interface - the ASA should be able to ping this interface.
The ASA encryption domain must be between the spoke and hub tunnel source addresses.

asa# sh crypto ipsec sa
interface: outside
    Crypto map tag: VPN_CMAP, seq num: 701, local addr: 13.164.30.6

      access-list VTC1-PE01_ROSNEFT-GW01 extended permit ip host 13.164.30.7 host 192.168.1.2
      local ident (addr/mask/prot/port): (13.164.30.7/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (192.168.1.2/255.255.255.255/0/0)
      current_peer: 13.161.4.151

      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
      #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
      #pkts compressed: 0, #pkts decompressed: 0