Firewall or ISP?

[gvaughn2246]

We have an ASA 5515 with dual ISP providers (Comcast & AT&T) set up. We have made no changes in regard to the Comcast interface nor the AT&T interface for about a year. All of sudden yesterday we were getting intermittent disconnects and reconnects (about 2-3 minutes up &5-10 minutes down) from our Comcast interfaces. Since there have been no changes made to firewall config and our Windstream connection was fine, I assumed it was on the ISP's end. They are telling me it's a firewall issue. I understand their reasoning however I'd like to get your opinion on the matter.

Here's the troubleshooting steps I've taken far; Comcast is saying everything is fine because if they take my public IP (ex; xxx.xxx.xxx.100) and set on another device like my laptop, everything works fine and does not drop, it's as soon as I plug it into the firewall when it starts dropping in/out. So, I double-checked everything NAT rules, ACL, route maps, etc. Everything looks fine and nothing's changed as expected. I went back to comcast, and they asked me to try another IP in the block, so I changed .100 to .101 and it worked perfect, no drops and consistent. But when I change back to .100 the disconnect issues immediately resume. I go back to comcast, and they tell me it must be a hardware issue with my firewall. I tell them that is highly unlikely as I have 2 firewalls stacked for failover and chances of both of them going out with the exact same issue is highly unlikely.  They are still sticking with the issue being on my end. So, I tried restoring my firewall to a known working date, and the exact same issue starts happening (note there have been almost no config changes for a year). 

Essentially this xxx.xxx.xxx.100 address will not hold a consistent connection, only on the firewall.  

I am at a loss at this point and go home for the night. The next morning, I come in and everything's working fine now with that address, again no config changes have been made for about a year. 4 hours after I left it the .100 address stabilized and has been up since then. This is great but I want to know why this happened to make sure it doesn't happen again or if the issue is even on my end. What do y'all think? I'm leaning more to layer 3 on ISP end now, but I'm not sure because of the laptop test.

[Flavio Miranda]

Hi,

  Probably anyone here already got in the situation of talk to an ISP, they deny any problem on their side and later on the problem is gone. They probably have found the problem after you complain but they will never tell you that someone there made some s***. 

 This is the standard all over the world. 

[MHM Cisco World]

sorry can you summary your issue?

[gvaughn2246]

Essentially the public xxx.xxx.xxx.100 address provided to me by my ISP will not hold a consistent connection on the firewall. However, if I set my laptop to xxx.xxx.xx.100 it works fine. This morning I came into the building and everything was now working properly. This makes me think it was really an ISP issue all along, however, comcast is still pointing the finger on our end. If it is on our end, I want to make sure it doesn't happen again

[MHM Cisco World]

can you share the 
show interface <outside> <<- of ASA

[gvaughn2246]

At the moment everything is working now

Interface Ethernet1/4 "COMCAST", is up, line protocol is up

  Hardware is EtherSVI, BW 1000 Mbps, DLY 10 usec

        Full-Duplex(fullDuplex), 1000 Mbps(1gbps)

        MAC address xxxxxxxxx, MTU 1500

        IP address xxx.xxx.xxx.xxx, subnet mask xxx.xxx.xxx.xxx

        637395 packets input, 96132600 bytes, 0 no buffer

        Received 176183 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        487320 packets output, 424235131 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops

  Traffic Statistics for "COMCAST":

        645297 packets input, 85322436 bytes

        499835 packets output, 433460082 bytes

        160226 packets dropped

      1 minute input rate 164 pkts/sec,  25965 bytes/sec

      1 minute output rate 233 pkts/sec,  317813 bytes/sec

      1 minute drop rate, 1 pkts/sec

      5 minute input rate 34 pkts/sec,  5516 bytes/sec

      5 minute output rate 46 pkts/sec,  56554 bytes/sec

      5 minute drop rate, 2 pkts/sec

[MHM Cisco World]

        160226 packets dropped <<- this only 

please check show asp drop <<- share here please 

[gvaughn2246]

Everything is still up and running consistently, the issue has not occurred for over 48 hours now. I will post results if it occurs again.

[MHM Cisco World]

thanks, 
MHM

[paul driver]

Hello
Any possible chance of ip address duplication, whenever that public .100 was being used by another device your network dropped out
Does your logging buffer show anything?
Take a snapshot of your arp entries and the next time it does this cross check it again.

[gvaughn2246]

That was my first thought except on the providers end, I don't even know how another device on our network could grab that IP address, but if it happens again, I will look into that.

As far as logging goes, we were able to reproduce the issue consistently by switching the Ip address back and forth, the only notable logging message we got when the downtime occured was "Routing failed to locate next hop for icmp from COMCAST: xxx.xxx.xxx.100/0 to COMCAST: Random IP". As far as I could tell these messages didn't really help point us in any direction, moreso just indicated the already known failure.

[MHM Cisco World]

Routing failed to locate next hop for icmp from <<- 
are you using any IP SLA for ISP ?

[gvaughn2246]

We are for both ISP providers.

[MHM Cisco World]

you meaning Yes ?
if Yes what is the target IP for IP SLA monitor ?
are this IP is reachable via both ISP ?

[gvaughn2246]

Yes.

So, the comcast is being tracked with AT&T serving as our backup. 4.2.2.1 is the target IP.  

Currently everything is up and is reachable, I am unsure at the time of failure if it was, however. If it happens again, I will check and respond back to this reply.