Filtering IPsec Traffic

[nsti_0682]

I have a very strange issue that I can't explain that I was hoping for fresh ideas.

Scenario - We have 100 spokes, all have 881's with dual DMVPN tunnels to two different Data Centers. The DMVPN hub routers site behind ASA's.

Issue - One of the spokes (let's call it site 83) can not ping the DMVPN hub tunnel IP.

Troubleshooting done - 

• Spoke 83's DMVPN tunnel is up and EIGRP neighborship is up (i.e. multicast packets are making it across the tunnel)
• Spoke 83 can SSH to the hub tunnel IP (i.e. SSH packets are making it across the tunnel)
• Spoke 83 can telnet to a server within the Data Center successfully (i.e. telnet packets are making it across the tunnel)
• ICMP traffic from the hub is reaching the spoke (verified via packet capture)
• ICMP traffic from spoke to spoke tunnel IP is successful but not to LAN side IP's
• Spoke 83 can SSH other spoke LAN side IP's
• There are no access-lists on the tunnel interfaces
• The encrypted ICMP traffic is leaving the Spoke 83 router but never reaching the Data Center ASA
  • I verified this by sending ICMP packets at 1000 bits and see them leaving the spoke 83 router (via packet capture) but they never arrive at the ASA's outside interface (verified via packet capture)
• ICMP packets to the internet egress locally at spoke 83 successfully
• ICMP packets to the DMVPN hubs external IP are successfully
• The Data Centers are geo diverse and are with two completely different Data Center vendors
• This issue occurred after the ISP replaced the cable modem at spoke 83

Question - At this point it seems pretty clear that the issue lies with the ISP cable modem but I can't explain how/why something is able to filter out only the ENCRYPTED ICMP packets yet other encrypted packets have no issue reaching the destination. Any ideas guys?

[Mohammed al Baqari]

Did you try to ping with size of 1320-bytes.

ping ip x.x.x.x size 1320

They might be putting filters based on low size packets

[nsti_0682]

Mohammed,

Yes I've done a ping sweep starting with 36 byte all the way to 1360.

[Mohammed al Baqari]

What is the output of show ip route x.x.x.x (x.x.x.x should be the IP of
the hub tunnel interface). Then get show ip cef x.x.x.x

What happens we you try traceroute to the hub IP

Also, show ip nhrp brief (what do you see for the mapping of the hub
interface)

[nsti_0682]

Mohammed,

The output of 'show ip route x.x.x.x' and 'show ip cef x.x.x.x' are what would be expected (the tunnel interface and hub tunnel IP.) The output of 'show ip nhrp brief' shows the correct mapping. Regarding the traceroute, keep in mind that the packets are leaving the Cisco router properly as I verified via packet capture so there's no issue on that device. 

[nsti_0682 · Accepted solution]

*UPDATE*

As expected after the ISP replaced the modem the issue is now resolved. This is concerning because somehow the 'faulty' modem was able to differentiate between IPsec encrypted ICMP packets and other IPsec encrypted traffic. Big brother or some other malicious agent on the modem?