Solved: puzzle: when the ISP refuses to use arp requests...

busitech · ‎10-13-2015

I have been having trouble doing something that should be incredibly simple, and now that it has proven difficult, it's a puzzle waiting to be solved.

The ISP has given us a few static addresses to use. We have a Cisco 2800 series router / firewall device. I'd like to assign the first address to the router, and use the rest with NAT / PAT.

With a routed subnet, this would be easy. However, the catch is that the static addresses are within one huge /23 subnet.

Any of the addresses work fine when assigned to the router as the primary address. The other addresses do not work when configured as a secondary address on the same interface as the primary, or with a NAT mapping, unless the second address has been recently used as the primary address of the router, or a physical host in the DMZ. If the address is removed as the primary address, and added to the NAT / PAT configuration we want, an "activated" address will work for many hours and then die, usually within 24 hours.

I decided to log arp packets on the outside interface for a few days. I discovered that the default gateway on the ISP side will never send an arp request for an address that has died off, even though I'm sitting on the other side of the 'net pinging the address. It seems obvious to me that the arp cache has expired on their end, and neither the router nor the ISP is announcing or requesting via arp.

After having had (way too many unproductive) conversions with the ISP, they continually refuse to make any changes on their end. I suggested a static arp entry for our mac address, or to start sending arp requests. I also can't get enough visibility into their configuration.

Does anyone know how to bridge arp announcements (gratuitous arp) from inside to outside, or how to get the 2800 IOS to generate gratuitous arp for the addresses that are used in NAT?

milan.kulik · ‎10-15-2015

Hi,

I found in my lab (an 1812 router running IOS 12.4(15)T9) that

router# clear arp secondary_ip_address

triggers gratuitous ARPs sent for ALL router interface IP addresses.

So running some script which would clear arp on your router periodically might be a workaround for your problem?

Best regards,

Milan

View solution in original post

milan.kulik · ‎10-16-2015

Hi,

does that mean the ISP router is refusing gratuitous ARPs from your secondary IP addresses?

And you would need to push your router to send ARP request for the default GW IP address with the secondary IP address used as a source IP of that request packet?

It's getting really mad...

The only way I see it to create a script which would swap the IP address of your router WAN port from the primary IP to each of the secondary IP addresses and finally return to the primary.

But that would mean an outage in your client communication periodically, I'm afraid.

This is really crazy and you should discuss with your ISP what kind of service they are providing to you!

Best regards.

Milan

View solution in original post

Masoud Pourshabanian · ‎10-13-2015

Have you tried NAT without setting that IP as a secondary address? You do not need to set that IP on the interface when you are mapping your local IP to that IP.

busitech · ‎10-13-2015

Yes I tried both ways.

Masoud Pourshabanian · ‎10-14-2015

In cisco IOS, ARP entries exprie after 4 hours and Cisco sends request before any entries expire.

I think if ARP was the problem, It would happen after 4 hours, not after 18 hours or more.

I guess your router might be causing the problem. it might remove the IP assigned to the NAT internally after some while so your ISP does not have any idea about that IP to send ARP request. Does you router get overwhelmed with NAT translation?Try to upgrade your IOS.

Masoud

busitech · ‎10-14-2015

In Cisco IOS, the arp timeout is configurable per interface:

cisco2851(config-if)# arp timeout 120

Four hours is simply the default value.

Upgrading software is a good idea. I am currently running 15.1(4)M9 which is only one version behind. There is always a chance of a bug in any release...

Masoud Pourshabanian · ‎10-15-2015

You are right, but your ISP is in charge of that now. Sometimes, downgrading might have better result. Are you doing source NAT or destination NAT?

Morne Vermeulen · ‎10-14-2015

Hi,

As far as I know, ISP's don't like adding static mac entries because of the obvious security risks. MAC spoofing is so easy these days, you don't even have to enter the command line on windows to change a mac. If they tap your line (which in itself has a very low chance of happening, but still possible), whoever the spoofer is will see all traffic on your internet line.

How is the NAT'ing set up? Is your internal range consumed through an access list and natted that way? Or is it the normal nat inside/outside setup running?

Side note: Still new to networking, take my advise with a grain of salt.

Thanks & Regards,
Morne

Morne Vermeulen · ‎10-14-2015

ALSO: If they don't make use of arp requests, maybe implementing a routing protocol through to your ISP would be an idea?

If ARPs are blocked, maybe OSPF hello packets or so isnt?

busitech · ‎10-14-2015

Routing protocols is a good idea, and on this front I've already tried RIP and BGP. I haven't done anything with OSPF however.

Jon Marshall · ‎10-14-2015

The ISP is not doing their job properly.

They should either -

1) assign you a subnet and you use one of the IPs for the outside interface and the rest for NAT, no need for secondary IPs.

The ISP then needs to arp for all IPs and you need proxy arp enabled on your outside interface

or

2) assign you two ranges, one for the link between your router and them and the other you can just use for NAT, again no need for secondary IPs.

Then they should simply have a route for the subnet not assigned to the link pointing to the outside interface IP of your router.

If the ISP is refusing to do either then you need a new ISP because it should not be up to you to issue gratuitous arps to them.

Jon

busitech · ‎10-14-2015

Since the addresses work when directly assigned to a device in the DMZ, they believe they have done their job. To obtain a subnet from them, we would have to fill out subnet justification forms, and pay an additional monthly fee. The business owner decided to try to make this work, and therein lies the puzzle. : )

Jon Marshall · ‎10-14-2015

Can you clarify the first bit about the DMZ ?

By DMZ do you mean between the router and the ISP ?

Jon

busitech · ‎10-14-2015

Sure, if I place a Windows PC on a switch outside the 2851, hung directly from the ISP's optic modem, the Windows PC can be assigned any of the "static" addresses that we've been allocated (essentially no more than taken out of their DHCP rotation).

Jon Marshall · ‎10-14-2015

So the ISP must be sending arps for those IPs assigned to you.

Or do you initiate the connection from that PC ?

Jon

busitech · ‎10-14-2015

Workstations and even the router itself will send an arp announcement for their primary address, and trigger the activation of the address, causing it to work for a while. For a limited time, the ISP gateway will arp to refresh its cache.

However, if the device does not send an arp announcement periodically, this process shuts down and the address dies off. This description is based on experience, testing with addresses in different configurations, and watching the debug arp log on the router on the outside interface.

The only addresses I can keep going are those assigned as the primary on the interface, or on a physical workstation. The NAT / PAT address destinations and secondary addresses both will never "prime the pump". If I change the primary address of the router temporarily, then move it to a NAT destination, the address works perfectly for about 18 hours, then dies.