Packet drops between ASA and ISP Modem

[robertotron007]

Hello All,

I am hoping someone could help me out with an issue that we have with dropped packets. Cisco TAC for our ASA has determined it is not an issue with our firewall and that it is an issue with the ISP. So here is the issue.

We have a /29 block of IPs. I need to utilize port forwarding for port 443 for three different servers. From what I understand, I can do this with a single public IP. Since we have 5 usable public IPs, I am trying to set up forwarding on the ASA. Now, the forwarding sort of works. I can get packets to the destinations on all three servers from the outside with NAT statements. But the problem is that I get dropped packets somewhere past the ISP. I think it is between the ISP and the ASA.

For example... i am sitting at my desk internally and pinging google. All is fine. Once I try to use one of the other services using port 443, I started to drop pings to google. Which means everyone accessing the internet internally is getting cur off the internet. To proves this, a capture was done on the ASA. We can see the pings from my desk leaving the ISP interface on our ASA but not coming back.. sporadically.

From what I read online this is because of duplicate ARP entires. Each additional public IP that I am mapping is sharing the same MAC address of the ASA interface. So when the ISP modem tries to send packets to me, there is some confusion.

Now I explained this to the ISP and sounded like they never heard of this before. I figured this would be something common but maybe not. I asked if they could make static arp entries for the additional IP addresses and to change the MAC. After a week they told me they made them but it did not work. Then the ISP told me they made the static ARP entries but to the same MAC address the ASA interface... which defeats the purpose of making them. I told them again they had to just change a character in the MAC address. They said they weren't sure if they wanted to do that and told me they can sell me another block of IPs.

Can someone shed some light for me here and tell me the proper procedure for what I am trying to do. Maybe even educate me on how situations like they are normally handled.

Thank you!

[Markus Benz]

Hi,

I don't think this is a problem between ASA and ISP. Many IP addresses may have the same MAC address, that should be no problem. I don't fully understand where your desk is, but I guess it is on the internal side of the ASA (same side as the servers). In that case I would say it is much more a configuration problem on the ASA. Probably in the NAT or routing area.

If you access your web-services internally, do you use the private or the public IP to access it? I would recommend to use the private one. That maybe solves your problem.

If you use the public which is actually a NATed IP on the ASA, you might run into NAT problems.

Regards,
Markus

[robertotron007]

In order to access our internal servers internally with the public DNS name, Cisco TAC had me set up the following policies: 

nat (Main,Main) source dynamic any interface destination static 1.1.1.3 EXCH-443
nat (Main,Main) source dynamic any interface destination static 1.1.1.4 RDS-443
nat (Main,Main) source dynamic any interface destination static 1.1.1.5 FTP2

Without those, I had to use the internal IP to access these services.

[johnlloyd_13]

hi,

did you check/ask your ISP for speed/duplex setting?

i had an ASA deployment before where 10M speed only works.

try bypassing your ASA and directly assign your PC with one of the public IPs and test from there.

check the status of your /29 IPs if it's clean/white listed on the internet.

as a last resort, ask your ISP to change your /29 range.

[robertotron007]

Yes speed and duplex are fine.

I connected a laptop to our modem and tested with two different IPs (one at a time) and there were no issues. The issue only occurs when i am trying to access two IPs over the same connection at the same time so this getting it to fail with the laptop / modem scenario doesn't happen.

[Markus Benz]

ok.. understand...

Did it work before? When you accessed the Web-Services with private IP addresses?
If yes, why do you want to change it to public? I would always recommend you to user split DNS, meaning use the private IP's from inside and the public IP's from outside.

I guess you're using a software release higher then 8.3, correct?
Otherwise accessing NAT services from inside with public IP is not supported and you need to upgrade first.

Are you in the same network (1.1.1.x)? Or is this a DMZ and you're in a different network.
If you're in the same network, do you have "same-security-traffic permit intra-interface" configured or not?

Do you see any firewall logs during the moment the ping to internet stops?

I am not 100% sure the NAT configuration is correct as it is right now.
Could you try something like this

object network internal
 range [your internal network]
object network external
 host [IP address of your WAN interface]
object network server-internal
 host [server internal IP address]
object network server-external
 host [server external IP address]
nat (internal, internal) source dynamic internal external destination static server-external server-internal

Regards,
Markus

[robertotron007]

I am not sure it ever worked before. Our RDS server always had issues over port 443 but it doesnt get used much. It was until I moved the RDS to a DMZ, the FTP, and had another network using port 443 that the problem got real bad.

Sorry, I was using ASA 9.1.2 and upgraded to 9.3.2. I connected our old Watchgaurd firewall and the same issue happens there.

Local network: 192.168.0.0/24 (where my desk is located)

DMZ: 172.160.0.0/24

No matter if I am going from WAN to LAN (accessing FTP or 443 externally) or WAN to DMZ (accessing FTP or 443 externally) from a different IP than what the Local network is using, I am dropping packets past the ASA.

The 1.1.1.0/29 is my public block assigned from the ISP.

[Richard Burts]

If the packet capture shows packets going out but no responses coming back then it sure sounds like the problem is not on the ASA but is somewhere in the ISP or in the Internet. Just to be sure that I understand correctly what you are telling us - when you send packets with the source address as the current Public IP on the ASA then everything works, all the time, but when you send packets with source address other addresses in the /29 then sometimes it works but some packets are getting dropped?

A couple of thoughts occur to me:

- Do we know if is only some addresses in the /29 or all other addresses in the /29 that have this issue? (if you change the address translation/forwarding for the troubled server to other addresses in the /29 do they all display this symptom?) 

- When you did the test with the laptop were you using the addresses that are now being used for the server that has the problem?

- Would it be possible to set up a test in which you change the ASA Public IP to be one of the other addresses in the /29?

HTH

Rick

[robertotron007]

Thanks Rick, you are correct. But it's not that it sometimes work. As I am pinging google from my desk in the internal network and I start using anything to do with those other IPs... whether I set the DMZ to use one of them as the source address to the internet or if I access one of those other IPs from say an internal FTP from my home PC, those packets I am sending to google from my desk start to drop sporadically until I stop accessing whichever services on those other IPs. lol I hope that makes sense.

The FTP example... we have 10mb upload. If I set the FTP to work on our main IP then I can get 1.2Mbps. If I use one of those other IPs... i get around 100kbps but its jumps all over and everyones connection to the internet gets choppy or I lose those pings to google sporadically.

I have test with the other 5 usable IP addresse and the same issue happens.

When I tested with the laptop, I tried three different IPs and they were all fine. Because I was only using one IP at a time.

I can change the interface to 1.1.1.6 for example and see what happens.

But what I am trying to do should be possible on the ASA right? Use all 5 usable public IPs on one ASA interface?

[Markus Benz]

as stated before, I am not sure NAT is configured correctly.

I don't think you should see packets leaving the WAN interface if the have source internal and destination DMZ. And if they leave the WAN interface, the destination IP and MAC is probably the ASA itself. So the packet will never reach the ISP. I really don't think it is a problem on the ISP side.

Did you check the NAT configuration, And did you compare it to what I've sent you before?

From my point of view the packet should be send from internal to DMZ directly, without leaving the WAN interface.

Original packet should look like this (recieved on the internal interface):
Src: <private IP of connecting host>
Dst: <public IP of server>

And after NAT the packet should look like this (sent out on DMZ interface):
Src: <WAN IP of the ASA, which is configured for dynamic NAT>
Dst: <private IP of server>

Regards,
Markus

[robertotron007]

Hello Markus,

Well, I would say lets not focus on that right now unless those NAT statements are a problem with the actual issue I am having. Let's put the DMZ aside too.

Two servers on the internal network / same local subnet that both require port 443 forwarded to them from the internet. One is a mail server where a NAT policy is set up from the public IP 1.1.1.3. That is also the IP that would show up when I go to "whatsmyip.org" from my desk. When I set up another NAT policy to tell the ASA to send traffic on port 443 to an internet RDS via IP 1.1.1.4 and I start to access that RDS from the outside... I start to lose packets that leave the ASA.

I tested that by watching my pings from leaving the ASA via the ISP connection but not coming back.

[Richard Burts]

This is puzzling and I do not understand what is going on. You certainly should be able to use all 5 Public IPs that are assigned to you (with 1 as the ASA interface address and the others used for Address Translation).

Part of me wonders if the issue might be load related. As multiple servers are being used could the increased load be triggering something that interferes with the packets? But the original poster seems pretty sure that they symptoms are related to the other addresses being used. So it is a puzzlement. Perhaps the next step is for the original poster to post the interface and address translation parts of the configuration. Perhaps the output of show xlate might also shed a little light on what is going on.

HTH

Rick

[robertotron007]

Were not too busy here so I dont think its load related.

Txt file attached.

[Karsten Iwen]

1) You should revert all ARP/MAC manipulations that you did. It's not unlikely that they cause the problems. You have a pretty standard scenario where all should work without changing anything in that area.

2) How did you test it to determine where the packets get dropped?

A way to find out what happens is to do a capture on the public interface of the ASA or even better on a switch with a SPAN-port between the ASA and the ISP-"modem" (which in fact is probably a router).

[robertotron007]

Cisco TAC did a packet capture watching ICMP packets incoming and outgoing the ASAs public interface. I was doing a steady ping from my desk to google and then I would try to access some of the services from the other public IP addresses. When I saw the pings drop from my desk, we would check the ASA packet capture and see the packets leaving the ASA but not coming back.