Ping blocked after device establishes connection from downtime.

[KGrev]

Hi,

I'm having an issue on our network. We have an snmp page that polls many devices for information. If a device is offline for maybe 10 to 30 minutes, the snmp page continues to ping the device waiting for it to reestablish connection. When the device comes back on, the snmp page will still fail to ping it. However, any other device, even in the same subnet, can ping the device and log into its webpage. We can create this issue with other devices though. From a desktop we can repeat continuous pings to a laptop on a vpn. When we turn off the vpn and allow the pings to fail for a few minutes, we can reestablish the vpn but the pings will still fail even though other devices can ping the device. Our only way to fix this has been to disable the snmp polling for an hour and restart the services. Pings will pass after that.

Has anyone else ran into this issue where "when a route has been determined to fail, it continues to fail until it is given up on for a long time"? Maybe a switch/router/or ASA is convinced the route is bad until it forgets about it?

Thanks for any help!

[Richard Burts]

That is a strange issue. It might be helpful to see the results of traceroute when the remote device is working and then the output when the remote device is not responding.

[KGrev]

@Richard BurtsThank you for your response.

Here is a traceroute with a working ping, then a failed ping and trace after the same device was disconnected from VPN then reconnected after 30 minutes. The device retained the same ip from the Firewall.

Reply from X.Y.242.46: bytes=32 time=59ms TTL=125
Reply from X.Y.242.46: bytes=32 time=63ms TTL=125
Reply from X.Y.242.46: bytes=32 time=65ms TTL=125
Reply from X.Y.242.46: bytes=32 time=48ms TTL=125
Reply from X.Y.242.46: bytes=32 time=133ms TTL=125

Ping statistics for X.Y.242.46:
Packets: Sent = 13, Received = 13, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 42ms, Maximum = 147ms, Average = 72ms
Control-C
^C
C:\Users\User.sa>tracert X.Y.242.46

Tracing route to X.Y.242.46 over a maximum of 30 hops

1 <1 ms 7 ms <1 ms X.Y.6.254
2 1 ms 1 ms 1 ms X.Y.5.1
3 1 ms <1 ms <1 ms FW1-INT [X.Y.5.34]
4 108 ms 60 ms 78 ms X.Y.242.46

*** FAILED PING****

C:\Users\User>tracert X.Y.242.46

Tracing route to X.Y.242.46 over a maximum of 30 hops

1 9 ms <1 ms <1 ms X.Y.6.254
2 1 ms 1 ms 1 ms X.Y.5.1
3 1 ms <1 ms <1 ms FW1-INT [X.Y.5.34]
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.

[Richard Burts]

What is the device at X.Y.5.34? 

I am curious about the 5 responses of Reques timed out. Was it really 5 responses and then it stopped? Or were there more responses and you did not bother to copy them since they were the same?

[KGrev]

@Richard Burts 

X.Y.5.34 is the ASA/FW. Its a cisco 5585

And yes the pings will continue to fail for a long time, even though other PC in my same subnet can ping what I can't. I cut off the rest of the pings for the forum post.

[KGrev]

@Richard BurtsCould this be some form of ASP learning the route is "bad" while the vpn is disconnected then holding onto that information? If I do a capture for  "capture asp-drop type asp-drop" I get hits for my ip trying to reach the vpn device. Would it show in this capture otherwise?

Thanks for your help

[Richard Burts]

Thanks for the additional information. It is interesting that 5.34 is the ASA. It certainly suggests that something on the ASA is causing the behavior. Don't have enough information to know whether it is ASP related or not.

[KGrev]

A little more info. The vpn device or the "unreachable device" can always ping back to the deivce that cant ping.

Also when looking through the asp-drops, occasionally I will see "icmp request Drop-reason: (punt-no-mem) Punt no memory.

I saw that there exists a cisco but and one user added route lookup to a nat rule. I didnt have a nat rule for this specific intereaction but I added one and added route lookup. No change yet.

[KGrev]

Good morning,

I'd like to add some more information to this if I can.

My issue seems to be completely restricted to icmp. While the issue is occurring and I cannot ping the distant device and the distant device can successfully ping me, I can navigate to the file structure of the distant device and host a fake website from that device that I can also pull at the same time that I cannot ping it.

Thanks for everyones help thus far.

[Richard Burts]

Thanks for the additional information. It clearly establishes that the problem is not basic IP connectivity but seems to be protocol dependent. That suggests some policy involvement, which would seem to point toward your ASA as the issue.

[KGrev]

@Richard Burts100% agree with you. I just have no idea how to sniff this one out. While its happening  basic informational logs don't show any traffic at all. I'll need to look at debugging logs I assume but I'm not sure which ones to go for.

[KGrev]

@Richard BurtsHere is some output from checking policy flow.

-------------------------------------------------------------------

FW1-EXT/pri/act# show run policy-map
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
no tcp-inspection
policy-map type inspect ipsec-pass-thru IMP_Max
parameters
esp
ah
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect esmtp
inspect ftp
inspect h323 h225
inspect h323 ras
inspect ip-options
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect sqlnet
inspect sunrpc
inspect tftp
inspect xdmcp
inspect icmp
inspect icmp error
class class-default
set connection timeout idle 1:00:00
set connection decrement-ttl
user-statistics accounting

------ICMP----

show service-policy flow icmp host A.B.6.9 host A.B.242.45

Global policy:
Service-policy: global_policy
Class-map: inspection_default
Match: default-inspection-traffic
Action:
Input flow: inspect icmp
Class-map: class-default
Match: any
Action:
Output flow: Input flow: set connection timeout idle 1:00:00
set connection decrement-ttl
Output flow: user-statistics accounting

------TCP-------

show service-policy flow tcp host A.B.6.9 host A.B.242.45

Global policy:
Service-policy: global_policy
Class-map: class-default
Match: any
Action:
Output flow: Input flow: set connection timeout idle 1:00:00
set connection decrement-ttl
Output flow: user-statistics accounting

---------------------------------------------------

Looks like when using ICMP for this path, it does fall into a different inspection compared to general tcp traffic.

( Class-map: inspection_default
Match: default-inspection-traffic)