Reduced Bandwidth on ASA 5516

[Garrett Hensley]

I'm looking for some troubleshooting tips on an ASA 5516. The outside interface is connected to a Cable Modem with a 300Mbps/25Mbps advertised speed. Typically I get around 325 down and 30 up. After a period of time, maybe 30 minutes I haven't timed it. The upload speed with drop to around 7~8Mbps. I don't have hardly any traffic on the ASA and the download speeds remain good. The ASA is cruising at about 2% cpu and 1342MB of memory (out of 8192MB). 

The inside interface is connected to a Cisco 3750 stack. 

I don't see any interface errors on the ASA or the 3750.

If I connect my laptop to another interface on the modem, the speed is 7~8Mbps as well....BUT!, if I disconnect the ASA, the upload speeds return to around 29~30Mbps. Then, after a period of time drop back down.

I'm not sure where to troubleshoot to eliminate the ASA and say the problem is in the modem.

[balaji.bandi]

If I connect my laptop to another interface on the modem, the speed is 7~8Mbps as well....BUT!, if I disconnect the ASA, the upload speeds return to around 29~30Mbps. Then, after a period of time drop back down.

as per the information above for me looks like ISP modem or ISP side issue for now

Remove ASA from the path totally and test, if the results same after some time the upload speed go down, you need to raise an ticket with ISP and sort this issue, 

if that speed is consistent, only you see this observation then we can look later ASA config and your setup.

[Garrett Hensley]

@balaji.bandi unfortunately, to do this requires downtime. Perhaps I can schedule an after hours period to have the network disconnected an only my laptop online. What bothers me is that I can disconnect the ASA and the speed goes back to normal. I already called the ISP and they are less than helpful in diagnosing the problem. 

[balaji.bandi]

When you get chance, introduce any switch between provider and ASA, so you can connect PC for testing when the things go slow to test yourself, where the problem occuring.

[Aref Alsouqi]

Very interesting! A few troubleshooting steps come to my mind:

a) what the ASA logs show while you have the download speed dropped? anything interesting?

b) I would try to reload the ISP device while the issue is happening and see if that restores it, if so, probably it is something on the ISP side to check and fix

c) ask the ISP to check if there are any CRCs on their CPE device, if so, then you would need to check the physical cable and maybe hardcode the speed/duplex on both sides. If this doesn't help, maybe you can move the connection to different ports as one or the two ports might be faulty

[Garrett Hensley]

Thank you for the reply.

A) I haven't seen anything that jumps out at me in the ASA logs. I did verify we do not have any rate limiting on the ASA. I do get the occasional scanning log, but from what I understand that is informational only.

B) I did reload the modem and the issue did not resolve, or it returned before I was able to check the speed again. I can temporarily resolve the issue by disconnecting the patch cable or resetting the ASA interface with a shut/no shut. Though that just proves stopping traffic resolves it, not necessarily where the issue is.

C) I wish my ISP would be more proactive in helping me troubleshoot. I have called them twice and they only say "everything looks good here". But all they do is ping the modem and measure latency. I am going to have them replace the modem since they will do that for free. If the send a technician they charge if they find no issue. I was hoping to eliminate the ASA for sure before doing that.

Thanks again,

Garrett

[Aref Alsouqi]

Hi Garrett, at this point probably you have an endpoint or more on your LAN that is/are consuming a lot of bandwidth, maybe they are infected. What you can try to do to get closer to the root cause would be to shutdown the inside interface whilst the issue exists and see if that fixes the issue. If so, then you need to find out the nodes that are taking down the Bandwidth. A way to do that would be via NetFlow, but if you don't have it enabled, then you can do it by raising the logs level on the firewall, or by creating an ACL on the inside interface with a permit ip any any with logging enabled on that rule.

[Garrett Hensley]

Thank you for the reply. When the event occurs I have monitored bandwidth through ASDM. Both the inside interface and the outside interface have negligible traffic. When I do a speed test, the bitrate matches the Speedtest very closely. If I had a high bandwidth user I would think I would see the interface bandwidth be higher.

[Aref Alsouqi]

This is becoming even more interesting! I would try to change the interfaces on both the ASA and the ISP CPE to a different one and the cable and see if that makes any difference. If that doesn't help probably I would ask the ISP to change their device, and I also would evaluate upgrading the ASA to the latest recommended release. Finally, please share the ASA sanitized config for review.

[Garrett Hensley]

I used a different cable and a different port on the ISP equipment when connecting with my laptop and produced the same results. I have a ticket in now to replace the ISP equipment. When I called they said there was a technician in the area working and that it may be related to the problem. Why they can't see what the technician's orders are I cannot fathom, but they couldn't tell me. So, there is a slim chance it's upstream, but the fact and interface reset fixes it makes me think otherwise. Anyhow, they are supposed to replace the equipment today too, so we will see. Of course, when I called the issue wasn't present.

Garrett

[Aref Alsouqi]

Curious to know the outcome  

[Garrett Hensley]

I wanted to post an update to this problem...not quite a solution just yet.

I had the ISP replace their equipment with no resolution. I installed PRTG to collect data to look for correlations of packet discards or errors. Again, no solution. HOWEVER, I connected directly to the modem using a local address so I could check out the modem's config and look at logs. I did a speed test and viola! The speed test on my laptop was almost 30Mbps while my desktop connected through the ASA was about 8 Mbps! BUT WAIT! THERE'S MORE!

Then, I changed my laptop to have one of our static addresses. With a static address on the laptop the speed test showed 8 Mbps! I did a trace route to 8.8.8.8 and the third hop is different when using a static vs using the internal address (essentially mimicking what a non static customer would be). So, my current theory is that there is sometimes congestion beyond my modems gateway since the traffic is routed differently when using a static vs using the modems ip address. I'm going to do some more testing today and contact the ISP again.

Garrett