03-02-2023 04:32 PM - edited 03-02-2023 04:36 PM
Hi,
in our network we have several ASR9K (9906 to be precise) and on most of them in order to manage remote sites, we create a BVI interface that will act as default gateway and bridge it together with the l2 transport interface that delivers the traffic from the optical ring.
We've recently installed a new ASR9K that is configured as following:
0/RSP0/CPU0 A9K-RSP5-SE(Active)
0/RSP1/CPU0 A9K-RSP5-SE(Standby)
0/FT0 ASR-9906-FAN
0/FT1 ASR-9906-FAN
0/0/CPU0 A9K-8HG-FLEX-SE
0/1/CPU0 A9K-8HG-FLEX-SE
0/2/CPU0 A9K-8HG-FLEX-SE
0/FC0 A99-SFC3-T
0/FC2 A99-SFC3-T
0/FC4 A99-SFC3-T
0/PT0 A9K-DC-PEM-V3
This router is running eXR 7.3.2 with the following packages:
asr9k-xr-7.3.2 version=7.3.2 [Boot image]
asr9k-mpls-x64-2.0.0.0-r732
asr9k-mpls-te-rsvp-x64-2.1.0.0-r732
asr9k-mgbl-x64-2.0.0.0-r732
asr9k-mcast-x64-2.0.0.0-r732
asr9k-ospf-x64-2.0.0.0-r732
asr9k-services-x64-1.0.0.0-r732
asr9k-k9sec-x64-2.2.0.0-r732
asr9k-optic-x64-1.0.0.0-r732
asr9k-bng-supp-x64-1.0.0.0-r732
asr9k-li-x64-1.1.0.0-r732
asr9k-9000v-nV-x64-1.0.0.0-r732
asr9k-eigrp-x64-1.0.0.0-r732
asr9k-isis-x64-1.1.0.0-r732
asr9k-m2m-x64-2.0.0.0-r732
asr9k-bng-x64-1.0.0.0-r732
asr9k-os-supp-64-4.0.0.4-r732.CSCwc53271
asr9k-common-pd-fib-64-1.0.0.2-r732.CSCvz95640
asr9k-gcp-fwding-64-1.0.0.10-r732.CSCvz95640
kernel-image-3.14.23-wr7.0.0.2-standard-3.14.p1-r0.1.r732.CSCwb68549.xr
asr9k-iosxr-infra-64-2.0.0.19-r732.CSCwa40057
asr9k-iosxr-fwding-64-1.1.0.21-r732.CSCwc63374
asr9k-bgp-x64-2.0.0.2-r732.CSCwa74538
asr9k-bng-x64-1.0.0.5-r732.CSCwb30446
asr9k-ospf-x64-2.0.0.2-r732.CSCvz57398
asr9k-mcast-x64-2.0.0.2-r732.CSCwb68540
asr9k-base-64-5.0.0.10-r732.CSCwc17209
asr9k-spirit-boot-64-1.0.0.1-r732.CSCwc40419
asr9k-fwding-64-4.0.0.18-r732.CSCwb82470
On this router we've noticed that we fail to ping the IP addresses of all the BVI interfaces we created on the router (ping to local loopback interfaces and local physical interfaces works fine) as well as all as the devices that are on the same network of the BVI)
On this router we have several BVI interface configured, for this scenario 2 of them are relevant: BVI42 and 87. This is the configuration:
interface BVI42
ipv4 address 10.242.13.1 255.255.255.128
!
interface BVI87
ipv4 address 10.242.118.177 255.255.255.240
When I try to ping from the router the IP address of BVI42 the ping fails, turning on debugging for the ping this is the message in te log:
RP/0/RSP0/CPU0:2023 Mar 3 01:18:15.882 : ipv4_io[341]: ipv4_icmp_error_trigger: type: 3, code: 3
RP/0/RSP0/CPU0:2023 Mar 3 01:18:15.882 : ipv4_io[341]: ipv4_icmp_error_trigger : interf addr is NULL, if hdl 0x6160 idb exists: 0
RP/0/RSP0/CPU0:2023 Mar 3 01:18:15.882 : ipv4_io[341]: ipv4_icmp_error_trigger : ipv6 encap case: 0
RP/0/RSP0/CPU0:2023 Mar 3 01:18:15.882 : ipv4_io[341]: Preroute for pak 0x63c0118b to i/f BVI87 requested
RP/0/RSP0/CPU0:2023 Mar 3 01:18:15.882 : ipv4_io[341]: Sending pak 0x63c0118b to NetIO towards i/f BVI87
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.070 : ipv4_io[341]: ipv4_icmp_error_trigger: type: 3, code: 3
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.070 : ipv4_io[341]: ipv4_icmp_error_trigger : interf addr is NULL, if hdl 0x6160 idb exists: 0
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.070 : ipv4_io[341]: ipv4_icmp_error_trigger : ipv6 encap case: 0
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.070 : ipv4_io[341]: Preroute for pak 0x63c0118b to i/f BVI87 requested
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.070 : ipv4_io[341]: Sending pak 0x63c0118b to NetIO towards i/f BVI87
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.141 : ipv4_io[341]: ipv4_icmp_error_trigger: type: 3, code: 3
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.141 : ipv4_io[341]: ipv4_icmp_error_trigger : interf addr is NULL, if hdl 0x6160 idb exists: 0
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.141 : ipv4_io[341]: ipv4_icmp_error_trigger : ipv6 encap case: 0
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.141 : ipv4_io[341]: Preroute for pak 0x63c0118b to i/f BVI87 requested
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.141 : ipv4_io[341]: Sending pak 0x63c0118b to NetIO towards i/f BVI87
RP/0/RSP0/CPU0:2023 Mar 3 01:18:17.340 : ipv4_io[341]: ipv4_icmp_error_trigger: type: 3, code: 3
There's probably something that is not immediately apparent here, but why from the debug it seems the route is trying to forward the packet out of BVI87 when I am trying to ping the IP of BVI42?
Ping from the router will also fail if I try to ping any of the remote sites.
However, if from any other router of the network I try to ping either the BVI address or the address of one of the devices, the ping will be successful. I am wondering if this could be related to the output of the "Show adjacency bvi-fia location 0/RSP0/CPU0" CLI for this router:
Service card for BVI interfaces on node 0/RSP0/CPU0
MPLS packets: Slot NONE VQI NONE
Non-MPLS packets: Slot NONE VQI NONE
----- Service Card List -----
0/RSP0/CPU0 VQI: Not Applicable
0/RSP1/CPU0 VQI: Not Applicable
0/0/CPU0 VQI: Not Applicable
0/1/CPU0 VQI: Not Applicable
0/2/CPU0 VQI: Not Applicable
0/3/CPU0 VQI: Not Applicable
0/FC0 VQI: Not Applicable
0/FC1 VQI: Not Applicable
0/FC2 VQI: Not Applicable
0/FC3 VQI: Not Applicable
0/FC4 VQI: Not Applicable
As I said earlier, we have this kind of topology on other ASR9k that run the same packages as this one and where local ping will work, so I"ve tried to run this CLI there as well and the output is different:
Service card for BVI interfaces on node 0/RSP0/CPU0
MPLS packets: Slot 0/1/CPU0 VQI 0x0
Non-MPLS packets: Slot 0/1/CPU0 VQI 0x0
----- Service Card List -----
0/RSP0/CPU0 VQI: Not Applicable
0/RSP1/CPU0 VQI: Not Applicable
0/0/CPU0 VQI: 0x20
0/1/CPU0 VQI: 0x0
0/2/CPU0 VQI: Not Applicable
0/3/CPU0 VQI: Not Applicable
0/FC0 VQI: Not Applicable
0/FC1 VQI: Not Applicable
0/FC2 VQI: Not Applicable
0/FC3 VQI: Not Applicable
0/FC4 VQI: Not Applicable
The difference in hardware configuration is that on this router 0/0/CPU0 is a A9K-48X10GE-1G-SE and 0/1/CPU0 a A9K-8HG-FLEX-SE.
Could this be the cause of the local failure of the ping? Any advice on how to troubleshoot this further?
Thank you
03-02-2023 05:57 PM
Hi @ThomasD86
Is the BVI up ? In order for the BVI to be up it needs to be map to L2VPN BD, and at least 1 Attach Circuit (AC) (l2transport subinterface) should be up.
I did read your statement about ping to BVI from remote locations works, and ping to hosts behind the BVI / BD works fine.
show run l2vpn bridge....
show run interface bvi xxx
show interface bvi xxx
show route 10.242.13.1
show route 10.242.118.177
show l2vpn bridge....
show uidb data location 0/0/CPU0 bvi xxx egress
show uidb data location 0/0/CPU0 bvi xxx ingress
Diego
03-03-2023 01:37 AM - edited 03-03-2023 01:44 AM
Hi Diego,
thanks for your reply. The BVI interfaces are indeed working, they're bridged together with an AC configured on a bundle made out of two interfaces from LC 0/1/CPU0(All BVIs interfaces on the router have their L2 AC configured under this bundle).
All the bridge groups, BVI and L2 AC have this configuration (of course with the appropriate values changed)
l2vpn
bridge group 42
bridge-domain 42
mtu 9180
interface Bundle-Ether12.42
!
routed interface BVI42
RP/0/RSP0/CPU0:ASR9k#sh run int Be12.42
Fri Mar 3 09:53:58.287 CET
interface Bundle-Ether12.42 l2transport
encapsulation dot1q 42
rewrite ingress tag pop 1 symmetric
mtu 9198
!
interface BVI42
description *** L3 MANAGEMENT***
ipv4 address 10.242.13.1 255.255.255.128
The output of the show uidb data is in spoiler tags as it is quite long:
egress:
Ingress:
03-03-2023 03:47 PM
Can you share, or look yourself for following fields in LC1 since is the one with the AC (BE with member only in LC1 you mention)
This is from the outputs of LC0
UIDB_IPV4_ENABLE_FLAG 1
UIDB_IPV4_ICMP_PUNT_FLAG 1
UIDB_IPV6_ENABLE_FLAG 0
UIDB_IPV6_ICMP_PUNT_FLAG 0
Do we see our own BVI "up" for ARP?
show arp bvi xxx
If you ping your own BVI IP Address those that work?
If you have 2 RPs we can try to ping from the Standby RP? is it cXR or eXR? what version?
Regards
Diego
03-03-2023 04:08 PM
Hi Diego,
pinging the BVI address from the router or even just trying to ping any device in the ARP table for that BVI will always fail when we ping directly from the router, it does work however when we try the same pings from a remote router. I can't quite understand this why locally generated traffic is getting blackholed while traffic that is generated remotely will go through.
I am not so sure how the router handles traffic generated locally especially something that has no ingress interface per se such as ICMP vs traffic that comes in from a remote node.
Looking through the BST, I found a couple DDTS that seems to closely match what I am experiencing:
CSCwc13576 CSCwd34293
In both cases the bvi-fia adjacency output is the same and so is the output of the "show adjacency hardware bvi trace" CLI
We do have fpd auto-upgrade configured as before they go live the routers do get updated to eXR 7.3.2 and have several SMUs installed.
You think we could be hitting one of those two DDTS?
Thank you
03-06-2023 12:23 PM
:D, sure maybe, but what version and is eXR of cXR? Did u had any LC that was remove and never place back?
Diego
03-07-2023 05:12 AM - edited 03-07-2023 06:07 AM
Hi Diego,
it's eXR 7.3.2, unbeknownst to me a colleague had already open a SR to TAC and the engineer confirmed we're hitting CSCwd34293
Thanks for all the help!
03-07-2023 06:47 AM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide