WAN next hop not default gateway

[Steve D]

Hello,

My router's WAN IP is assigned via DHCP and is in subnet 45.44.42.64/26. My default gateway IP is 45.44.42.65 (same subnet). For some reason when I'm tracing the route to any WAN address instead of my WAN gateway being the next hop, I'm seeing 70.66.160.1 instead.

Can anyone explain how/why this is happening? With a bit of searching it seems that when my WAN interface arps for the WAN gateway IP, a MAC address is returned that isn't actually for my WAN gateway IP. Apparently this could be easier to configure for an ISP rather than setting up multiple router IP's, and also has the benefit of not consuming an IP in a subnet for the gateway. If this is the case, I don't understand why I get 45.44.42.65 as my gateway from DHCP, shouldn't I just get 70.66.160.1 as the gateway? Or is this to solve client router compatibility issues?

I should also mention this ISP is a reseller, so perhaps this is a typical configuration for a reseller ISP?

traceroute output to WAN gateway:

traceroute to 45.44.42.65 (45.44.42.65), 30 hops max, 60 byte packets
1 192.168.10.1 2.139 ms 1.979 ms 1.904 ms
2 70.66.160.1 10.141 ms 11.533 ms 12.826 ms
3 45.44.41.13 13.959 ms 18.173 ms 17.914 ms
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

Traceroute to another WAN address:

traceroute to 4.2.2.1 (4.2.2.1), 30 hops max, 60 byte packets
1 192.168.10.1 3.253 ms 3.216 ms 4.563 ms
2 70.66.160.1 12.990 ms 13.017 ms 13.003 ms
3 45.44.41.13 18.964 ms 18.924 ms 18.825 ms
4 66.163.72.10 24.711 ms 24.667 ms 24.676 ms
5 45.44.103.21 21.641 ms 21.563 ms 21.584 ms
6 45.44.104.25 20.609 ms 17.350 ms 20.373 ms
7 38.88.6.233 19.167 ms 11.431 ms 15.433 ms
8 154.24.48.217 15.520 ms 154.24.48.221 16.447 ms 16.336 ms
9 154.54.83.225 19.223 ms 154.54.24.137 18.805 ms 21.602 ms
10 * * *
11 4.69.152.207 139.988 ms 4.69.152.79 139.920 ms 138.902 ms
12 4.69.152.79 138.742 ms 4.69.152.143 138.631 ms 4.69.152.79 138.554 ms
13 4.2.2.1 135.000 ms 134.881 ms 128.021 ms

[Pawan Raut]

can you send me sh ip int brief and sh ip route output 

[milan.kulik]

sh arp  might be also helpful.

[Steve D]

I was trying to keep this as a high level exercise as I'm not using a Cisco router in this configuration yet, but I can include the same type of info from pfsense:

Arp table:

Interface	IP	MAC	Notes
LAN	192.168.10.1	52:54:00:1f:77:cd	Router LAN Int
LAN	192.168.10.104	c0:f2:fb:15:c9:48
WAN	45.44.42.65	00:01:5c:88:92:46	WAN Gateway
WAN	45.44.42.xx	52:54:00:62:c3:63	Router WAN Int
LAN	192.168.10.152	52:54:00:71:dc:a1
LAN	192.168.10.252	52:54:00:11:c5:a2
LAN	192.168.10.253	d0:50:99:25:72:fd

Routing table:

Destination	Gateway	Flags	Use	Mtu	Netif	Expire
default	45.44.42.65	UGS	3613	1500	em1
45.44.42.64/26	link#2	U	114903	1500	em1
45.44.42.xx	link#2	UHS	0	16384	lo0
127.0.0.1	link#6	UH	13182	16384	lo0
192.168.10.0/24	link#1	U	11759018	1500	em0
192.168.10.1	link#1	UHS	0	16384	lo0

Interfaces:

WAN Interface (wan, em1)

Status

up

DHCP

up

MAC Address

52:54:00:62:c3:63

IPv4 Address

45.44.42.xx

Subnet mask IPv4

255.255.255.192

Gateway IPv4

45.44.42.65

IPv6 Link Local

fe80::5054:ff:fe62:c363%em1

ISP DNS servers

127.0.0.1

192.168.10.252

192.168.10.253

MTU

1500

Media

1000baseT <full-duplex>

In/out packets

12733398/8454088

In/out packets (pass)

12733398/8454088

In/out packets (block)

276280/1

In/out errors

0/0

Collisions

0

LAN Interface (lan, em0)

Status

up

MAC Address

52:54:00:1f:77:cd

IPv4 Address

192.168.10.1

Subnet mask IPv4

255.255.255.0

IPv6 Link Local

fe80::5054:ff:fe1f:77cd%em0

MTU

1500

Media

1000baseT <full-duplex>

In/out packets

8583235/11759976

In/out packets (pass)

8583235/11759976

In/out packets (block)

5032/0

In/out errors

0/0

Collisions

0

[Richard Burts]

I understand your wanting to present this as a high level exercise, but unfortunately I believe that what you have is a low level issue. And we do not know enough to be sure that any answer that we give would be the correct answer. From what you have told us it is clear that DHCP provides one address as the gateway address and that when you arp for that address you receive a response, and that traffic forwarded using that MAC address does get forwarded. So fundamentally the network is not broken.

It seems to me that there might be a couple of things that might be the answer:

- the ISP device might have address 45.44.42.65 but be configured in bridge mode and the device that is doing the layer 3 forwarding is 70.66.160.1 and therefore is the device generating the response address for traceroute.

- it may be that for reasons that make sense to them the ISP has configured DHCP to provide 45.44.42.65 but has configured their device address as 70.66.160.1.

- when your device sends the arp request it receives a response giving a MAC address. But we do not know whether this response was a native arp response from an IP in the same subnet or whether this was generated by proxy arp on a device with an address in a different subnet.

HTH

Rick 

[milan.kulik]

Hi Rick,

I guess it could also be the ISP device using 45.44.42.65 as a secondary IP address but 70.66.160.1 as the primary one?

If proxy arp would be involved, there should be plenty of ARP entries visible on Steve's WAN interface using the same MAC address, I guess?

@Steve:

Do you see an ARP entry on your device for 70.66.160.1 using the same MAC address as   45.44.42.65? And any other entries using it?

Best regards,

Milan

[Richard Burts]

Milan

Having a secondary address is an interesting idea and certainly is possible.

Your comment about the number of arp entries would be quite correct if the default route pointed just at the outbound interface (ip route 0.0.0.0 0.0.0.0 Fast0/1) but not so much if the default route points at the next hop address (ip route 0.0.0.0 0.0.0.0 45.44.42.65) in which case it would just arp for that address and then forward all the outbound traffic using the MAC that it learned.

HTH

Rick

[milan.kulik]

Hi Rick,

you are right, as usually!

Looking to the output provided by Steve second time:

traceroute to 45.44.42.65 (45.44.42.65), 30 hops max, 60 byte packets
1 192.168.10.1 2.139 ms 1.979 ms 1.904 ms
2 70.66.160.1 10.141 ms 11.533 ms 12.826 ms
3 45.44.41.13 13.959 ms 18.173 ms 17.914 ms
4 * * *
5 * * *

Hop #2 is the ISP router.

If it had 45.44.42.65 configured as a secondary IP address, it would NOT forward the traceroute packets to another device 45.44.41.13, would it?

So it seems the ISP router is having some more specific route than 45.44.42.64/26 configured - possibly a single host 45.44.42.xx given to Steve's router (to make the returning traffic routed correctly)?

And proxy-arp enabled to let the customer traffic to reach the Internet?

Best regards,

Milan

[Richard Burts]

Milan

I agree with you that if the ISP had configured a secondary address that the ISP router at hop 2 should not have forwarded the packet to hop 3. So it seems that 45.44.42.65 is not on the ISP router and that the ISP router must be using proxy arp to respond to the arp request for the gateway router.

HTH

Rick

[Steve D]

Thanks for providing some clarity into how the network could be configured. I hadn't seen anything like this before, and its always good to learn something new.

Milan - the ARP table I posted above is complete, there were no entries for IP 70.66.160.1.

Thanks again gents,

Steve