puzzle: when the ISP refuses to use arp requests...

[busitech]

I have been having trouble doing something that should be incredibly simple, and now that it has proven difficult, it's a puzzle waiting to be solved.

The ISP has given us a few static addresses to use.  We have a Cisco 2800 series router / firewall device.  I'd like to assign the first address to the router, and use the rest with NAT / PAT.

With a routed subnet, this would be easy.  However, the catch is that the static addresses are within one huge /23 subnet.

Any of the addresses work fine when assigned to the router as the primary address.  The other addresses do not work when configured as a secondary address on the same interface as the primary, or with a NAT mapping, unless the second address has been recently used as the primary address of the router, or a physical host in the DMZ.  If the address is removed as the primary address, and added to the NAT / PAT configuration we want, an "activated" address will work for many hours and then die, usually within 24 hours.

I decided to log arp packets on the outside interface for a few days.  I discovered that the default gateway on the ISP side will never send an arp request for an address that has died off, even though I'm sitting on the other side of the 'net pinging the address.  It seems obvious to me that the arp cache has expired on their end, and neither the router nor the ISP is announcing or requesting via arp.

After having had (way too many unproductive) conversions with the ISP, they continually refuse to make any changes on their end.  I suggested a static arp entry for our mac address, or to start sending arp requests.  I also can't get enough visibility into their configuration.

Does anyone know how to bridge arp announcements (gratuitous arp) from inside to outside, or how to get the 2800 IOS to generate gratuitous arp for the addresses that are used in NAT?

[Jon Marshall]

Any IP used for general internet access from your users should be updating the ISPs arp cache continuously.

But any static NAT statements for access to internal servers would require some sort of arp from the ISP.

So basically the ISP assigns you extra IPs but you must tell them you are using them with the service you purchased off them (sorry I can be a bit slow on the uptake sometimes :-)).

I am not aware of a way to get a router to send gratuitous arps for IPs used with NAT which is really what you need.

Are you using the additional IPs for one to one mappings between public and private IPs or are you doing port translations ie. using one public IP for many private IPs on different ports ?

Jon

[busitech]

Honestly, if I could get either 1:1 or port mapping to work, I'd conform to either method that was compatible.  Unfortunately, I started this thread because it looks like a real workstation has an edge over the NAT implementation of IOS, and I'd like to be proven wrong in this assessment.

[Jon Marshall]

Well to be fair an end device and a router are doing very different things so I'm not sure one has the edge over another.

If you are doing one to one mappings then a rather messy workaround could be to set up a scheduled job on the end device to ping an internet IP.

As long as the ping ran before the arp entry timed out it would automatically update the ISP's arp cache.

It is by no means an ideal solution and wouldn't work with port mappings but at the moment I am struggling to think of anything else.

Jon

[busitech]

As screwy as it sounds, I can use telnet to port 25, using a 1:1 NAT inside address, see the packet arrive at my own mail server with tcpdump on Linux, from the correct address after NAT, and still not see the return traffic, nor an arp request.  It just doesn't activate the IP or wake up the ISP's network to the presence of my now very active address...

It has been suggested to place a Raspberry Pi in the DMZ running arpsend...  : )

[Jon Marshall]

Sorry I am not following.

Do you mean telnet from the internet to the public IP ?

I am just doing a quick lab to see if I can come up with anything but not sure what you mean by the above ?

Also when you refer to DMZ just to clarify you do mean the network between you and the ISP ?

Jon

[busitech]

I meant that I log into an internal machine running Linux, with a private address, that the router currently has in the NAT map for 1:1 translation, and I can see the connection on the other end, using the correct public IP address, and still not seem to update anything at the ISP by virtue of the connection just made.  The connection hangs due to the absence of the return traffic, just like if I try to ping from the outside to inside, using the same 1:1 NAT entry.  This surprised me, because the ISP just send the traffic out, but didn't let it traverse the network in the opposite direction.  The telnet session just hangs, waiting...

When I use traceroute to observe the path inward towards the 2851, I also notice that the last hop is not the same as our default gateway.  That doesn't mean it's not the same device, but that they have more going on than one would expect.

[Jon Marshall]

Well you learn something new everyday :-)

It was always my understanding that any IP traffic would reset the arp cache timer but apparently, after doing a quick lab and a bit of searching on this forum, with IOS that doesn't seem to be the case.

So ignore my previous solution because it wouldn't work.

Basically what you saw was correct behaviour ie. the ISP router still has to do an arp for the public IP and this is why it didn't work.

When I did the lab even though I did the ping from inside to outside the ISP router still sent an arp for the public IP.

So yes you either need something to do gratuitous arps or you need the ISP to arp for those IPs which with your service isn't going to work.

Sorry about the misleading information.

Jon

[busitech]

No worries, Jon!  So glad to have your comments.  Thanks!  I learned a lot during this exercise.

[busitech]

Yes, when I say DMZ I mean attached directly to the ISP equipment, outside our firewall, outside the router, as a sibling / peer of the router's outside interface.

[busitech]

...and by explaining their position I do not mean to agree with the ISP, I'm just bringing in the facts as they stand.  They may know full well their stance on the matter will force a given client to pay for the premium services.

[Morne Vermeulen]

Hi,

I hope my input helped in some way. As I stated, still new to networking and how it works. 

[busitech]

During testing, I've tried 1:1 static NAT, NAT with port mapping (PAT), NAT and PAT with route maps, NAT and PAT with access lists, static arp entries, and assigning secondary address.  None of these create the arp traffic the ISP is looking for to activate an address.

It appears the way the ISP is set up right now, the gratuitous arp response that a workstation sends when it boots up, in order to detect duplicate addresses that may exist (and warn the user) and announce it's presence on the subnet (and update the arp tables of adjacent nodes) is what the ISP uses to "turn on" an address.

I can take over any IP address I assign to the 2851...  Even those not assigned to us!  As a test, I turned on local-proxy-arp (turning on proxy arp for the whole subnet) which allowed the 2851 to answer the arp requests that I do see as a result of any recently released DHCP leases from other clients in the subnet (which happens during the phase out period, as shown in my debug arp logs) and then those unassigned addresses were sent to us fir the duration of the phase out period (as shown by my ping responses from across the 'net).

Hard-coding my router's MAC address on their side would be similar to what cable companies do with a residential cable modem...  Their gateway upstream from the customer site will only talk to the MAC address we call in with.  When we buy a new cable modem, they turn off one MAC address, and turn on the other.

It is my observation that the way it is now represents the security risk, and I shared that with the ISP yesterday.

What frustrates me is that I can't yet find a way to have the Cisco 2851 (a very nice piece of premium enterprise networking equipment) simulate what the primitive IP stack in Windows XP does when it starts up, on behalf of the public addresses specified as a NAT destination or even secondary IP address.  Seems like the Microsoft operating system has one-upped my Cisco gear, which is hard for me to swallow!

[milan.kulik · Accepted solution]

Hi,

I found in my lab (an 1812 router running IOS 12.4(15)T9) that 

router# clear arp secondary_ip_address

triggers  gratuitous ARPs sent for ALL router interface IP addresses.

So running some script which would clear arp on your router periodically might be a workaround for  your problem?

Best regards,

Milan

[busitech]

Hi Milan, I tried this immediately, and I was surprised when it failed to activate the secondary address.  I then went to compare the arp debug log, to find the difference between assigning a primary address on the interface (which activates the address every time), and clearing the arp on a secondary address.  Here's what I found when I assigned an address of .233 to the primary address of the interface:

*Oct 16 06:49:03.704: IP ARP: sent rep src X.136.223.233 0023.5e36.7e81,

                 dst X.136.223.233 ffff.ffff.ffff GigabitEthernet0/1

*Oct 16 06:49:03.712: IP ARP: creating incomplete entry for IP address: X.136.222.1 interface GigabitEthernet0/1

*Oct 16 06:49:03.712: IP ARP: sent req src X.136.223.233 0023.5e36.7e81,

                 dst X.136.222.1 0000.0000.0000 GigabitEthernet0/1

*Oct 16 06:49:03.716: IP ARP: rcvd rep src X.136.222.1 f866.f2d2.c8c0, dst X.136.223.233 GigabitEthernet0/1

Here's what clearing the arp on a secondary address looks like:

*Oct 16 06:44:18.060: IP ARP: sent rep src X.136.223.233 0023.5e36.7e81,

                 dst X.136.223.233 ffff.ffff.ffff GigabitEthernet0/1

So, it looks like the ISP is activating an address when it receives an arp request from the same address for the default gateway.

Shucks, I thought we had it.  At least I learned something from the test.

[milan.kulik · Accepted solution]

Hi,

does that mean the ISP router is refusing gratuitous ARPs from your secondary IP addresses?

And you would need to push your router to send ARP request for the default GW IP address with the secondary IP address used as a source IP of that request packet?

It's getting really mad...

The only way I see it to create a script which would swap the IP address of your router WAN port from the primary IP to each of the secondary IP addresses and finally return to the primary.

But that would mean an outage in your client communication periodically, I'm afraid.

This is really crazy and you should discuss with your ISP what kind of service they are providing to you!

Best regards.

Milan