packet capture IPSEC VPN tunnel traffic

[JamesS4]

Hi, I am attempting to do a packet capture of traffic going across an IPSEC VPN tunnel to troubleshoot an issue (see here if interested: https://community.cisco.com/t5/routing/ipsec-tunnel-only-works-one-way/m-p/3876972#M316944), but none of it is showing in the capture for some reason.  

I read in a different thread to permit 'esp' traffic as this is the packet type going through the tunnel, but it has not resolved anything.  If I set up the capture on the inside interface I can capture traffic, but I cannot capture anything directly on the VPN tunnel interface itself for some reason (which is essential for debugging the issue).  I wonder if I should not be monitoring GI0/0/0, but rather there is some way to monitor the tunnel itself directly?

Config is as follows:

ip access-list extended BUF-FILTER
permit ip any host 198.198.4.51

permit ip any host 198.198.4.52

permit ip any host 198.198.4.53

permit ip host 198.198.4.51 any

permit ip host 198.198.4.52 any

permit ip host 198.198.4.53 any

permit esp any host 198.198.4.51

permit esp any host 198.198.4.52

permit esp any host 198.198.4.53

permit esp host 198.198.4.51 any

permit esp host 198.198.4.52 any

permit esp host 198.198.4.53 any

monitor capture buffer BUF filter access-list BUF-FILTER

monitor capture point ip cef POINT Gi0/0/0 both

thanks!!!

[Giuseppe Larosa · Accepted solution]

Hello JamesS4,

the ESP packets are exchanged between the two tunnel endpoints

>> We have IPSEC tunnel on outside Internet interface GI0/0/0 from our public address 38.38.38.98 to vendor side 198.198.4.38, which is working fine.

so you would need ACL lines like

ip access-list extended BUF-FILTER

permit esp host 38.38.38.98 host 198.198.4.38

permit esp host 198.198.4.38 host 38.38.38.98

You can also use the

show  crypto ipsec sa

that provides counters for encrypted /decrypted packets with separate sections for each line of the VPN_ACL that defines the interesting traffic.

This is probably the most useful tool as encrypted packets have user packets inside but they are encrypted.

Hope to help

Giuseppe

[JamesS4]

Hi Giuseppe,

Thanks for the help yet again.  This is a little troubling what you are suggesting, since the packet capture will not allow me to distinguish what I am looking for.

Basically, I want to be able to definitively see that I am receiving traffic from 198.198.4.53 to 10.10.10.50 via the VPN tunnel rather than the Internet and that we are also sending return traffic through the tunnel, un-NATed in both directions. 

How can I conclusively show this is occurring?

[Giuseppe Larosa · Accepted solution]

Hello JamesS4,

try to post

show crypto ipsec sa

so that we can see if there is a section for each line of ACL VPN_ACL.

if there is with separate counters for encrpyt / decrpyt you should be fine.

Hope to help

Giuseppe

[JamesS4]

There is an entry, which is good, I think...  but it doesn't give me the detail I want, e.g. for inbound traffic comes through the tunnel, is the return traffic also going out through the tunnel?

protected vrf: (none)
local ident (addr/mask/prot/port): (10.10.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (198.198.4.53/255.255.255.255/0/0)
current_peer 198.198.4.38 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 11, #pkts encrypt: 11, #pkts digest: 11
#pkts decaps: 3522, #pkts decrypt: 3522, #pkts verify: 3522
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 38.38.38.98, remote crypto endpt.: 198.198.4.38
plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0/0
current outbound spi: 0xE6BEE1BE(3871269310)
PFS (Y/N): N, DH group: none

inbound esp sas:
spi: 0xC0D61316(3235255062)
transform: esp-256-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2087, flow_id: ESG:87, sibling_flags FFFFFFFF80004048, crypto map: VPN
sa timing: remaining key lifetime (k/sec): (4608000/26632)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xE6BEE1BE(3871269310)
transform: esp-256-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2088, flow_id: ESG:88, sibling_flags FFFFFFFF80004048, crypto map: VPN
sa timing: remaining key lifetime (k/sec): (4608000/26632)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

[Giuseppe Larosa]

Hello JamesS4,

the interesting lines are

local ident (addr/mask/prot/port): (10.10.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (198.198.4.53/255.255.255.255/0/0)
current_peer 198.198.4.38 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 11, #pkts encrypt: 11, #pkts digest: 11
#pkts decaps: 3522, #pkts decrypt: 3522, #pkts verify: 3522

There are only 11 packets sent to 198.198.4.53 and 3522 received

In a working scenario the first pkts encaps should increment also when the connection is started from the remote site.

If this counter does not increment during an attempt to connect from remote site it is a confirmation return traffic is not going via IPSec ( it is an indirect info but it is meaningful)

Hope to help

Giuseppe

[JamesS4]

Yes it seems indirect deduction is as good as we can hope for. 

The question now is why is this happening??  =(

[JamesS4]

what i do not understand is why do we see this traffic at all in the capture?

38.38.38.98 -> 198.198.4.53 0 BE TCP

shouldn't this NAT not be occurring based on the NAT ACLs we have setup?

p7router1#sh ip nat trans
Pro Inside global Inside local Outside local Outside global

tcp 38.38.38.98:30001 10.10.10.50:30001 198.198.4.53:21459 198.198.4.53:21459

[Giuseppe Larosa]

Hello JamesS4,

the ip nat translation that you see is derived from the following statement

ip nat inside source static tcp 10.10.10.50 30001 38.38.38.98 30001 extendable

tcp 38.38.38.98:30001 10.10.10.50:30001 198.198.4.53:21459 198.198.4.53:21459

Have you got a different internal host in network 10.10.10.0/24 to make the tests instead of focusing on this 10.10.10.50 host?

The packet capture may be related to the nat static statement but in this case the involved port is 30001 and not the telnet port that is 23.

To which TCP port is the remote site attempting to telnet to?

Hope to help

Giuseppe

[JamesS4]

attempting to telnet to 30001!!!  I think (hope) that is the problem?!

[Giuseppe Larosa]

Hello JamesS4,

if the other side has always tried to telnet to TCP 30001 this is the root cause of the problem as the static NAT is triggered when using this specific TCP port.

Ask them to use a different TCP port or provide them a different target in 10.10.10.0/24 internal network.

Hope to help

Giuseppe

[JamesS4]

We opened an unused port on 10.10.10.50 and asked them to try it.  fingers crossed!

[JamesS4]

I selected two of Giuseppe's replies as solutions, since they are both correct.  

1) VPN tunnel packet capture can only help to detect traffic travelling across the tunnel endpoints.  There isn't a way to directly capture traffic from device endpoints.

2) looking at ipsec details shows endpoint connection stats, which aren't as ideal as a direct capture, but still provides the most  useful and relevant information