cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
7624
Views
10
Helpful
14
Replies

ASA5505 - outbound traffic ceases even though port is up

Darren Coleman
Level 1
Level 1

Hi,

I've had a Cisco ASA 5505 firewall connected to a cable modem (Virgin Media, UK) for the past 3 years.  In the last 6 months or so I have noticed that the ASA would drop the outside (internet) connection intermittently, usually at least once every 1-2 weeks - the interface still shows as being up but no traffic crosses it, and computers on the inside network abruptly lose internet connectivity.  Rebooting the ASA or administratively shutting down the interface and bringing it back up again would cure the problem straight away until the next time it happens.

In the last couple of days however despite nothing having been changed in the configuration the frequency of this connection drop has increased to the point where I would lose access to the internet within an hour of rebooting the ASA.  It does not seem to matter whether or not there is traffic currently going out or not, inside computers just appear to suddenly lose internet connectivity.

I have tried the following without success:

1) I completely wiped the configuration (configure factory-default)

2) I changed the port the cable modem was connected to (eth0/0 -> eth0/7, changing switchport vlan accordingly)

I thought perhaps 2) had fixed it but it lasted a whole 2 hours before I woke up this morning to find that none of the internal equipment had internet access despite the fact eth0/7 was showing as up/up in ASA CLI.

This morning I manually set the eth0/7 port to "speed 10" (10Mbps, full duplex).  It was previously set to be auto-negotiation (default) on both speed and duplex.  As of this post it has managed to keep the outside connection up for 3 hours - but I'm not optimistic that it is fixed.

Interface counters have never shown any collisions, errors, etc - only packets input and output as expected.

Since the problem persisted across ports (eth0/0 -> eth0/7) I'm wondering whether or not the problem could either be faulty memory, or some kind of speed/duplex incompatibility between the cable modem and ASA. Would a duplex mismatch manifest in sporadic total connection dropouts though?

Thanks in advance for any help

Daz

1 Accepted Solution

Accepted Solutions

Hello Daz,

The debugs clearly show that the issue is with the next hop device and not with the ASA.

Before shutting the interface, we cleary see an arp request going out of the ASA but there is no response from the next hop device.

Once we bounce the interface, the behaviour expected from the connected device is to clear arp entry and create fresh enteries. Thus, after shutting down the interface, we see a response coming back from the next hop and arp entry gets build.

Hope this helps. Please reply back if you need any further assistance.

Regards,
Chirag
P.S.: Please mark this thread as answered if you feel your query is answered. Do rate helpful posts.

View solution in original post

14 Replies 14

danilew
Level 1
Level 1

Do all computers loose internet connectivity or just some?  Can you post a log showing the traffic dropping along with your config?

Everything behind the firewall on the inside network loses internet connectivity immediately.  There is no speed degradation or abrupt connection termination - it just appears as if no data is being received.  Eventually applications trying to connect to the internet say "timed out" or equivalent.

The last time this happened I was unable to ping an address (e.g. www.google.co.uk - 74.125.230.147) from the ASA CLI either, when prior to the connection drop I could.  Everything on the internal network is still able to access the ASA on its internal IP address.  I am confident that the cable modem is not faulty because its operation and logs do not suggest anything is happening to it, and rebooting the ASA or simply shutting down the outside interface and bringing it back up again immediately fixes the problem, until it happens again.

It has been several hours since I forced the port speed on eth0/7 to 10Mbps (from auto-speed, auto-duplex) and it has maintained the outside connection.  I am at work though so there won't be much if any traffic going across it.  I don't know yet whether this has actually made a difference since the dropouts seem to be so random, and prior to the ASA starting to drop the outside connection within an hour or so of being rebooted it would last for a week or so.

Have attached sanitised config.  I don't have a log of the connection dropping unfortunately.

Thanks for your help!

Outside connection has died again, so setting it to 10Mbit didn't fix anything.

(Unfortunately I'm at work and ASA is at home, so I cannot remotely restart it or get logs, etc)

Hi Daz,

What syslogs do you see being generated in ASDM when the problem is happening?

One possibility is an ARP problem between your outside interface and the next hop gateway. Have you contacted your ISP when the problem is happening to ask if they see any traffic leaving your network?

When the problem is happening, can you ping your default gateway provided by the ISP from the ASA (check the output of 'show route' to find out what the IP address is)? If you can't ping it, do a 'clear arp', repeat the ping again, and then check the output of 'show arp' on the firewall to see if you have a valid MAC address for the gateway's IP address.

-Mike

Hi,

I haven't yet contacted my ISP to see if they see any traffic leaving my network.

When the problem occurs and the outside interface isn't passing traffic I am unable to ping the next hop gateway.  I simply get ????? on the ping results.  I tried "clear arp" and then pinging again with the same result.  "show arp" just showed one entry, the inside computer that I was using to SSH to the ASA.

I have attached a bunch of output I took while the problem was in effect (i.e. no traffic going out of the outside interface).

Please note: the network this ASA is on is very simple - I have several computers connected to the switch on the ASA (5 total), the ASA is connected via an ethernet cable to the cable modem.  The cable modem has no user configuration, it is configured by the ISP via a DOCSIS configuration file it downloads.

Hi Daz,

You mentioned that you only see an entry in the ARP table for the client PC after clearing the ARP cache, but I see the gateway's entry in the output you attached:

outside 94.169.96.1 0030.b8d2.1450 191

If that output was taken before clearing the cache and the gateway's MAC address is really not showing up in the ASA's ARP table, then you'll definitely want to contact your ISP to find out why the gateway is going unresponsive. You can use 'debug arp' when this problem is happening to see if the ASA is sending/receiving ARP replies.

The logs you posted indicate that the ASA is building connections for outside hosts, so it would be a good idea to get your ISP on the phone during an outage so they can look at things on their gateway. You could also open a TAC case for this to get an engineer to help make sure everything is OK on the ASA at the time of the problem.

-Mike

Thanks Mike.

All of the attached files was output taken after the first time it had dropped the connection today and before I saw your post suggesting to "clear arp,  ping gateway, show arp".  When I did what you suggested - clear arp,  ping gateway, show arp - there was only 1 ARP line listed (the client PC)

The next time it happens I will use "debug arp" to see what the ASA is trying to do when the outside interface appears to be down.

For clarity, this is what happened when I followed your instructions:

1) Before typing clear arp the output had the same entries as the attached asm-arp.txt file above

2) I type clear arp, and then ping 94.169.96.1

3) There is no response from that IP (as per the output in asa-route.txt)

4) I type show arp, there is only one entry for "scion" - which is the client PC on the inside network that I am SSH'ing to the ASA from

Thanks again for your help

Ok it happened again, this time I switched debug arp on to see what was happening.

I have attached the file showing the output from this.

You can see that until I shutdown the interface and bring it back up again it just keeps repeating the same two lines:

arp-req: generating request for 94.169.96.1 at interface outside
arp-req: request for 94.169.96.1 still  pending

...after the interface is shut down and brought back up, I get:

arp-req: generating request for 94.169.96.1 at interface outside
arp-req: request for 94.169.96.1 still  pending
arp-send: arp request built from 94.169.96.79 001e.f715.7a75 for 94.169.96.1 at 12461760
arp-in: response at outside from 94.169.96.1 0030.b8d2.1450 for 94.169.96.79 001e.f715.7a75
arp-set: added arp outside 94.169.96.1 0030.b8d2.1450 and updating NPs at 12461800
arp-in: resp from 94.169.96.1 for 94.169.96.79 on outside at 12461800

..and internet connectivity resumes.

Hope this provides some more info.

Hello Daz,

The debugs clearly show that the issue is with the next hop device and not with the ASA.

Before shutting the interface, we cleary see an arp request going out of the ASA but there is no response from the next hop device.

Once we bounce the interface, the behaviour expected from the connected device is to clear arp entry and create fresh enteries. Thus, after shutting down the interface, we see a response coming back from the next hop and arp entry gets build.

Hope this helps. Please reply back if you need any further assistance.

Regards,
Chirag
P.S.: Please mark this thread as answered if you feel your query is answered. Do rate helpful posts.

Hi,

Sorry for the late update but I wanted to leave it over the weekend to run some more tests.

Having checked the logs for the cable modem I can see that there are errors showing that correspond to the times that the outside connection appears to drop, which coobberates the opinion that it is the "next hop" that is at fault, not the ASA itself...

Sun Mar 13 01:20:23 2011     Sun Mar 13 01:20:23 2011     Critical (3)      SYNC Timing Synchronization failure - Loss of  Sync;CM-MAC=00:22:68:f0:77:20;

CMTS-MAC=00:30:b8:d2:14:50;CM-QOS=1.1;CM-VER=3.0;
Sun Mar 13 01:20:27 2011     Sun Mar 13 01:20:27 2011     Warning (5)     Lost MDD Timeout;CM-MAC=00:22:68:f0:77:
20;CMTS-MAC=00:30:b8:d2:14:50;CM-QOS=1.1;CM-VER=3.0;
Sun Mar 13 01:20:49 2011     Sun Mar 13 01:21:18 2011     Critical (3)     No Ranging Response received - T3 time-out
Sun Mar 13 01:21:20 2011     Sun Mar 13 01:21:20 2011     Critical (3)     Ranging Request Retries exhausted
Sun Mar 13 01:21:20 2011     Sun Mar 13 01:21:20 2011     Critical (3)      Unicast Maintenance Ranging attempted - No response - Retries  exhausted
Sun Mar 13 01:21:39 2011     Sun Mar 13 01:21:39 2011      Warning (5)     MIMO Event MIMO: Stored MIMO=0 post cfg file  MIMO=-1;CM-MAC=00:22:68:f0:77:
20;CMTS-MAC=00:30:b8:d2:14:50;CM-QOS=1.1;CM-VER=3.0;

(the above is probably meaningless in the context of Cisco ASAs but I am including it above so that anyone with a similar problem might find it on a search engine)

Thanks for all the help, for the moment I will consider this matter resolved from the point of view of my Cisco equipment.

Hi Daz,

Thanks for the update. I am glad that the issue is resolved.

Regards,
Chirag

Hi Daz

I'm experiencing the exact same problem with Virgin cable modem.  Did you manage to properly fix the problem?

Many thanks

Ben

In a manner of speaking yes.  As per the replies in this thread the issue was not with my ASA but with Virgin Media's network.  Shortly after I reported the problem to them the problem went away and hasn't reoccured since.

Chirag

I have experiencing similar problem where a Cisco ASA 5510 drops connection intermittently while the outpound interface shows up. A "debug arp" when the problem is occuring shows the following

arp-req: generating request for at interface OUTSIDE
arp-send: arp request built from 2c54.2d0c.823e for at 46324020
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending
arp-req: generating request for at interface OUTSIDE
arp-req: request for still  pending

this happens untill i issue "clear arp" after which internet gets restored and this shows


arp-in: response at OUTSIDE from 4403.a7f9.7458 for 2c54.2d0c.823e
arp-set: added arp OUTSIDE 4403.a7f9.7458 and updating NPs at 46324210
arp-in: resp from for on OUTSIDE at 46324210
arp-send: sending all saved block to OUTSIDE at 46324210

I have sent this ISP and still say everything is ok on their side.

I have tried adjusting putting a static arp entry which makes things worse; have to reboot the firewall when connection drops

I have tried adjusting arp timeout from default of 14400 to 180 but same problem comes up.

i have also realized i am getting same arp output from internal hosts

arp-req: generating request for 192.168.0.44 at interface USERS

arp-req: request for 192.168.0.44 still  pending

arp-req: generating request for 192.168.0.44 at interface USERS

arp-req: request for 192.168.0.44 still  pending

arp-req: generating request for 192.168.0.44 at interface USERS

arp-req: request for 192.168.0.44 still  pending

arp-req: generating request for 192.168.0.44 at interface USERS

arp-req: request for 192.168.0.44 still  pending

arp-send: arp request built from 192.168.0.1 2c54.2d0c.823f for 192.168.0.35 at 999380

arp-req: generating request for 192.168.0.35 at interface USERS

arp-req: request for 192.168.0.35 still  pending

arp-req: generating request for 192.168.0.35 at interface USERS

arp-req: request for 192.168.0.35 still  pending

arp-req: generating request for 192.168.0.35 at interface USERS

arp-req: request for 192.168.0.35 still  pending

arp-send: arp request built from 192.168.0.1 2c54.2d0c.823f for 192.168.0.44 at 1001380

arp-in: request at USERS from 192.168.0.43 dc0e.a1ea.7953 for 192.168.0.1 ffff.ffff.ffff

arp-in: rqst for me from 192.168.0.43 for 192.168.0.1, on USERS

arp-set: added arp USERS 192.168.0.43 dc0e.a1ea.7953 and updating NPs at 1001430

arp-in: generating reply from 192.168.0.1 2c54.2d0c.823f to 192.168.0.43 dc0e.a1ea.7953

arp-send: arp request built from 192.168.0.1 2c54.2d0c.823f for 192.168.0.35 at 1003380

arp-req: generating request for 192.168.0.44 at interface USERS

arp-req: request for 192.168.0.44 still  pending

arp-req: generating request for 192.168.0.44 at interface USERS

arp-req: request for 192.168.0.44 still  pending

arp-req: generating request for 192.168.0.44 at interface USERS

arp-req: request for 192.168.0.44 still  pending

arp-req: generating request for 192.168.0.44 at interface USERS

arp-req: request for 192.168.0.44 still  pending

so am trying to figure out what exactly happens when i "clear arp" that brings connection back?


Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: