There is a new behavior introduced by CSCse46790 which should be introduced soon (5.1.2 and above) but we still need to understand how XR behaves in previous releases.
There are 2 scenarios where this problem comes into the pictures:
- Non local ARP entry.
- Forwarding entry of /32 (or /128 for IPv6) RIB routes overridden when there is a covering connected or attached route.
Let's have a look at an example for each problem.
1. Non local ARP entry.
IOS XR learns ARP entries for non directly connected IPv4 addresses by default and these ARP entries will break connectivity to the corresponding /32 CEF prefix.
Workaround: configure "arp learning local" on all layer 3 interfaces but it's only available since 4.3.4 through CSCty06696 or through a SMU in some earlier releases.
Lab example:
These 2 routers are running OSPF so that router1 learns the loopback of router2 as an OSPF route:
RP/0/RSP0/CPU0:router1#sh route 10.0.0.107 Routing entry for 10.0.0.107/32 Known via "ospf core", distance 110, metric 2, type intra area Installed Feb 10 22:52:40.812 for 01:27:00 Routing Descriptor Blocks 10.0.1.3, from 10.0.0.107, via BVI2 Route metric is 2 No advertising protos. RP/0/RSP0/CPU0:router1#
The CEF entry points at 10.0.1.3 as next hop accordingly:
RP/0/RSP0/CPU0:router1#sh cef 10.0.0.107 location 0/0/CPU0
10.0.0.107/32, version 1447, internal 0x4004001 (ptr 0xa4442b08) [1], 0x0 (0xa3d61eb0), 0x440 (0xa555b5f0)
Updated Feb 11 00:18:21.516
local adjacency 10.0.1.3
Prefix Len 32, traffic index 0, precedence n/a, priority 1
via 10.0.1.3, BVI2, 4 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0xa59a4b58 0x0]
next hop 10.0.1.3
local adjacency
local label 16008 labels imposed {None}
RP/0/RSP0/CPU0:router1#
Now... if for some weird reason router2 sends an ARP reply or request with its loopback address 10.0.0.107 as "sender protocol address", router1 will learn that address in its ARP table even though this is not a local address on the subnet:
RP/0/RSP0/CPU0:router1#sh arp location 0/0/CPU0 Address Age Hardware Addr State Type Interface 10.0.0.107 00:00:03 0001.0001.0001 Dynamic ARPA BVI2 10.0.1.1 - 4055.3912.ece5 Interface ARPA BVI2 10.0.1.2 00:00:00 f025.72a7.ac3d Dynamic ARPA BVI2 10.0.1.3 00:27:36 001d.4603.1f42 Dynamic ARPA BVI2
A normal router or host is not supposed to send such an ARP for a non local address but we've seen some types of routers doing that. Or when different subnets are configured on the same broadcast domain.
When there's an ARP entry, CEF tries to add a corresponding adjacency prefix (/32 host route) with that IP/MAC mapping. When there is an existing host route from the routing table (RIB) like the 10.0.0.107/32 OSPF route, that RIB prefix route is removed from the FIB to be replaced by the FIB adjacency prefix since FIB prefers the adjacency prefixes over the RIB prefixes due to CSCse46790. But the problem here is that the adjacency prefix 10.0.0.107/32 will be declared as unresolved because the IP address of the non local ARP entry is not covered by the connected subnet configured on the interface. Since the RIB host route has been removed and the FIB adjacency prefix is marked as unresolved, we don't have a forwarding entry anymore for the original /32 RIB route (10.0.0.107/32 in our example) and traffic for that destination might be dropped (it might not be dropped if there is a less specific prefix like a default route for instance which can forward the packets towards the destination).
RP/0/RSP0/CPU0:router1#sh cef 10.0.0.107
10.0.0.107/32, version 1447, internal 0x4004001 (ptr 0xada9bfb0) [1], 0x0 (0xad4579f8), 0x440 (0xae07e500)
Updated Feb 10 22:52:40.833
Prefix Len 32, traffic index 0, precedence n/a, priority 1
via 10.0.1.3, BVI2, 2 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0xae39e0e0 0x0]
next hop 10.0.1.3
local adjacency
local label 16008 labels imposed {None}
RP/0/RSP0/CPU0:router1#sh cef 10.0.0.107 loc 0/0/CPU0
0.0.0.0/0, version 1407, proxy default, internal 0x4000021 (ptr 0xa4442060) [1], 0x0 (0x0), 0x0 (0x0)
Updated Feb 6 11:35:42.788
Prefix Len 0, traffic index 0, precedence n/a, priority 3
via 10.48.32.1, 3 dependencies, recursive [flags 0x0]
path-idx 0 [0xa4445edc 0x0]
next hop 10.48.32.1 via 10.48.32.1/32
RP/0/RSP0/CPU0:router1#
We can see that the 10.0.0.107/32 prefix is missing from the CEF table on 0/0/CPU0 as we are now matching the default route 0.0.0.0/0. The prefix is actually unresolved:
RP/0/RSP0/CPU0:router1#sh cef unresolved detail location 0/0/CPU0
10.0.0.107/32, version 0, internal 0x6484003 (ptr 0xa4442b08) [1], 0x0 (0x0), 0x40 (0xa555c2b0)
Updated Feb 11 00:45:57.790
Prefix Len 32, traffic index 0, Adjacency-prefix, precedence n/a, priority 0
via 10.0.0.107, BVI2, 0 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0x0 0x0]
unresolved
local label 16008 labels imposed {None}
RP/0/RSP0/CPU0:router1#
There are 2 solutions:
- Configure "arp learning local" on all layer 3 interfaces when the router is running an IOS XR image which contains the fix for CSCty06696 (Add CLI option to handle out-of-subnet ARP requests ) so that the router will not learn non local arp entries (and hence not cause a problem with FIB adjacency prefixes being preferred over RIB routes).
- Upgrade to an IOS XR version with the fix for CSCse46790 (CEF prefers arp adjacency over RIB next hop ) where the default XR behavior will be changed to prefer RIB prefixes over adjacency prefixes (with a new config command "cef adjacency route override rib" to maintain the old behavior for those who have come to rely on it).
One of the 2 options should be enough to protect the router.
Another option if you don't have one of the 2 fixes is to identify the hosts sending the non local arp packets and fix their config or block them... but you won't really be protected for long.
2. Forwarding entry of /32 (or /128 for IPv6) RIB routes overridden when there’s a covering connected or attached route
When an IPv4 host or IPv6 host is supposed to be directly connected on an interface and we want to redirect traffic for that host through another interface by configuring a /32 IPv4 route or /128 IPv6 route, then this route may not be taken into account due to CSCse46790 (CEF prefers arp adjacency over RIB next hop) so that we may still forward traffic over the interface with the covering prefix instead of the other interface where we configured the /32 or /128 route to point to.
Example:
If router1 wants to reach 10.0.2.3, it would normally used the directly connected prefix 10.0.2.0/24 over BVI3.
But if we want to manually route traffic for 10.0.2.3 over BVI2, we would configure a static route like:
router static address-family ipv4 unicast 10.0.2.3/32 BVI2 10.0.1.3
This static route would look good on the RP/RSP:
RP/0/RSP0/CPU0:router1#sh route 10.0.2.3
Routing entry for 10.0.2.3/32
Known via "static", distance 1, metric 0
Installed Feb 11 01:14:46.207 for 00:21:48
Routing Descriptor Blocks
10.0.1.3, via BVI2
Route metric is 0
No advertising protos.
RP/0/RSP0/CPU0:router1#sh cef 10.0.2.3
10.0.2.3/32, version 1448, internal 0x4004001 (ptr 0xada98578) [1], 0x0 (0xad457098), 0x440 (0xae07e140)
Updated Feb 11 01:14:46.217
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via 10.0.1.3, BVI2, 2 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0xae39e0e0 0x0]
next hop 10.0.1.3
local adjacency
local label 16010 labels imposed {None}
RP/0/RSP0/CPU0:router1#
But on the linecard, the CEF adjacency prefix created from the ARP entry for 10.0.2.3 over BVI3 will take precedence over the RIB prefix from the static route for 10.0.2.3 pointing at BVI2 so forwarding will still be through BVI3:
RP/0/RSP0/CPU0:router1#sh cef 10.0.2.3 loc 0/0/CPU0 10.0.2.3/32, version 0, internal 0x4480001 (ptr 0xa444343c) [1], 0x0 (0xa3d649d0), 0x0 (0x0) Updated Feb 11 01:14:46.227 local adjacency 10.0.2.3 Prefix Len 32, traffic index 0, Adjacency-prefix, precedence n/a, priority 0 via 10.0.2.3, BVI3, 2 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 [0xa59a41b8 0x0] next hop 10.0.2.3 local adjacency RP/0/RSP0/CPU0:router1#
This can only be avoided by running an image with the fix for CSCse46790 (CEF prefers arp adjacency over RIB next hop ) where the default XR behavior will be changed to prefer RIB prefixes over adjacency prefixes (with a new config command "cef adjacency route override rib" to maintain the old behavior for those who have come to rely on it).
