We have an ASA 5515 with dual ISP providers (Comcast & AT&T) set up. We have made no changes in regard to the Comcast interface nor the AT&T interface for about a year. All of sudden yesterday we were getting intermittent disconnects and reconnects (about 2-3 minutes up &5-10 minutes down) from our Comcast interfaces. Since there have been no changes made to firewall config and our Windstream connection was fine, I assumed it was on the ISP's end. They are telling me it's a firewall issue. I understand their reasoning however I'd like to get your opinion on the matter.
Here's the troubleshooting steps I've taken far; Comcast is saying everything is fine because if they take my public IP (ex; xxx.xxx.xxx.100) and set on another device like my laptop, everything works fine and does not drop, it's as soon as I plug it into the firewall when it starts dropping in/out. So, I double-checked everything NAT rules, ACL, route maps, etc. Everything looks fine and nothing's changed as expected. I went back to comcast, and they asked me to try another IP in the block, so I changed .100 to .101 and it worked perfect, no drops and consistent. But when I change back to .100 the disconnect issues immediately resume. I go back to comcast, and they tell me it must be a hardware issue with my firewall. I tell them that is highly unlikely as I have 2 firewalls stacked for failover and chances of both of them going out with the exact same issue is highly unlikely. They are still sticking with the issue being on my end. So, I tried restoring my firewall to a known working date, and the exact same issue starts happening (note there have been almost no config changes for a year).
Essentially this xxx.xxx.xxx.100 address will not hold a consistent connection, only on the firewall.
I am at a loss at this point and go home for the night. The next morning, I come in and everything's working fine now with that address, again no config changes have been made for about a year. 4 hours after I left it the .100 address stabilized and has been up since then. This is great but I want to know why this happened to make sure it doesn't happen again or if the issue is even on my end. What do y'all think? I'm leaning more to layer 3 on ISP end now, but I'm not sure because of the laptop test.
Probably anyone here already got in the situation of talk to an ISP, they deny any problem on their side and later on the problem is gone. They probably have found the problem after you complain but they will never tell you that someone there made some s***.
This is the standard all over the world.
Essentially the public xxx.xxx.xxx.100 address provided to me by my ISP will not hold a consistent connection on the firewall. However, if I set my laptop to xxx.xxx.xx.100 it works fine. This morning I came into the building and everything was now working properly. This makes me think it was really an ISP issue all along, however, comcast is still pointing the finger on our end. If it is on our end, I want to make sure it doesn't happen again
At the moment everything is working now
Interface Ethernet1/4 "COMCAST", is up, line protocol is up
Hardware is EtherSVI, BW 1000 Mbps, DLY 10 usec
Full-Duplex(fullDuplex), 1000 Mbps(1gbps)
MAC address xxxxxxxxx, MTU 1500
IP address xxx.xxx.xxx.xxx, subnet mask xxx.xxx.xxx.xxx
637395 packets input, 96132600 bytes, 0 no buffer
Received 176183 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
487320 packets output, 424235131 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
Traffic Statistics for "COMCAST":
645297 packets input, 85322436 bytes
499835 packets output, 433460082 bytes
160226 packets dropped
1 minute input rate 164 pkts/sec, 25965 bytes/sec
1 minute output rate 233 pkts/sec, 317813 bytes/sec
1 minute drop rate, 1 pkts/sec
5 minute input rate 34 pkts/sec, 5516 bytes/sec
5 minute output rate 46 pkts/sec, 56554 bytes/sec
5 minute drop rate, 2 pkts/sec
Any possible chance of ip address duplication, whenever that public .100 was being used by another device your network dropped out
Does your logging buffer show anything?
Take a snapshot of your arp entries and the next time it does this cross check it again.
That was my first thought except on the providers end, I don't even know how another device on our network could grab that IP address, but if it happens again, I will look into that.
As far as logging goes, we were able to reproduce the issue consistently by switching the Ip address back and forth, the only notable logging message we got when the downtime occured was "Routing failed to locate next hop for icmp from COMCAST: xxx.xxx.xxx.100/0 to COMCAST: Random IP". As far as I could tell these messages didn't really help point us in any direction, moreso just indicated the already known failure.