New ISP causes some paths on IPv6 to become unavailable

[MP13]

Hi,
We are currently working to bring AS3223 live as a transit provider for IPv4 and IPV6 however when bringing IPv6 live we are running in to an issue where traffic via them becomes unreachable and doesn't leave our network. We've worked with them to establish that it's not an issue with their network and from our side too the configuration is the same as other connections we have.

I have included output of the route to Google which is one that we know is affected. What I have noticed is that the best path stays the same which is via a peering exchange however the source interface changes which is presumably why the traffic then stops working. 

I don't understand why the source interface changes and things then stop working though when bringing up AS3223. I've tried using update-source and ipv6 next-hop commands but this doesn't change things. 

Before:

sh bgp ipv6 unicast 2607:f8b0::/32
BGP routing table entry for 2607:F8B0::/32, version 86683745
BGP Bestpath: deterministic-med
Paths: (4 available, best #4, table default)
Multipath: eBGP
Not advertised to any peer
Refresh Epoch 1
174 6453 15169
XXXXXX::1C1:1 (FE80::2C1:64FF:FE5D:FEE0) from XXXXXX1::1C1:1 (XXXXXX.1.207)
Origin IGP, metric 9020, localpref 100, valid, external
Community: XXXXX:1003
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
174 6453 15169, (received-only)
XXXXXX::1C1:1 (FE80::2C1:64FF:FE5D:FEE0) from XXXXXX::1C1:1 (XXXXXX.1.207)
Origin IGP, metric 9020, localpref 100, valid, external
Community: 174:21100 174:22010
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
15169
XXXXXX::3B41:1 (FE80::8AA2:5EFF:FE16:887B) from XXXXXX::220A:2 (XXXXXX.225.231)
Origin IGP, metric 0, localpref 100, valid, external
Community: XXXXX:1002
unknown transitive attribute: flag 0xE0 type 0x20 length 0x18
value 0000 220A 0000 03E8 0000 0001 0000 220A
0000 03E9 0000 0002
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
15169
XXXXXX::3B41:1 (FE80::8AA2:5EFF:FE16:887B) from XXXXXX::220A:1 (XXXXXX.225.230)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: XXXXX:1002
unknown transitive attribute: flag 0xE0 type 0x20 length 0x18
value 0000 220A 0000 03E8 0000 0001 0000 220A
0000 03E9 0000 0002
rx pathid: 0, tx pathid: 0x0

sh ipv6 route 2607:f8b0::/32
Routing entry for 2607:F8B0::/32
Known via "bgp XXXXX", distance 20, metric 0, type external
Route count is 1/1, share count 0
Routing paths:
FE80::8AA2:5EFF:FE16:887B, TenGigabitEthernet0/0/0.91
MPLS label: nolabel
Last updated 1w4d ago

After:
sh bgp ipv6 unicast 2607:f8b0::/32
BGP routing table entry for 2607:F8B0::/32, version 86683745
BGP Bestpath: deterministic-med
Paths: (6 available, best #6, table default)
Multipath: eBGP
Not advertised to any peer
Refresh Epoch 1
3223 1299 15169
XXXXXX:4::4 (FE80::211:11FF:FE2A:7ECB) from XXXXXX:4::7 (XXXXXX.172.110)
Origin IGP, localpref 100, valid, external
Community: XXXXX:1004
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
3223 1299 15169, (received-only)
XXXXXX:4::4 (FE80::211:11FF:FE2A:7ECB) from XXXXXX:4::7 (XXXXXX.172.110)
Origin IGP, localpref 100, valid, external
Community: 3223:888
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
174 6453 15169
XXXXXX::1C1:1 (FE80::2C1:64FF:FE5D:FEE0) from XXXXXX::1C1:1 (XXXXXX.1.207)
Origin IGP, metric 9020, localpref 100, valid, external
Community: XXXXX:1003
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
174 6453 15169, (received-only)
XXXXXX::1C1:1 (FE80::2C1:64FF:FE5D:FEE0) from XXXXXX::1C1:1 (XXXXXX.1.207)
Origin IGP, metric 9020, localpref 100, valid, external
Community: 174:21100 174:22010
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
15169
XXXXXX::3B41:1 (FE80::8AA2:5EFF:FE16:887B) from XXXXXX::220A:2 (XXXXXX.225.231)
Origin IGP, metric 0, localpref 100, valid, external
Community: XXXXX:1002
unknown transitive attribute: flag 0xE0 type 0x20 length 0x18
value 0000 220A 0000 03E8 0000 0001 0000 220A
0000 03E9 0000 0002
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
15169
XXXXXX::3B41:1 (FE80::8AA2:5EFF:FE16:887B) from XXXXXX::220A:1 (XXXXXX.225.230)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: XXXXX:1002
unknown transitive attribute: flag 0xE0 type 0x20 length 0x18
value 0000 220A 0000 03E8 0000 0001 0000 220A
0000 03E9 0000 0002
rx pathid: 0, tx pathid: 0x0

sh ipv6 route 2607:f8b0::/32
Routing entry for 2607:F8B0::/32
Known via "bgp XXXXX", distance 20, metric 0, type external
Route count is 1/1, share count 0
Routing paths:
FE80::8AA2:5EFF:FE16:887B, TenGigabitEthernet0/0/0.91
MPLS label: nolabel
Last updated 1w4d ago

[Harold Ritter]

Hi @MP13 ,

> however when bringing IPv6 live we are running in to an issue where traffic via them becomes unreachable and doesn't leave our network.

The Google prefix is not through AS3223 and seems to be causing issues, which conflict with what you are stating here. Can you please give us more information about the issue.

> the best path stays the same which is via a peering exchange however the source interface changes

Can you please let us know what source interface you are referring to?

Are you conducting the test from the router itself?

Regards,

[MP13]

Hi Harold,

Apologies, I didn't check my output. What I thought I had seen at one point was that after I raised the session to AS3223 although the best path stayed the same it changed the below

From FE80::8AA2:5EFF:FE16:887B, TenGigabitEthernet0/0/0.91 to FE80::8AA2:5EFF:FE16:887B, TenGigabitEthernet0/0/0.89

However on doing it again just now to check it hasn't changed and has in fact stayed as FE80::8AA2:5EFF:FE16:887B, TenGigabitEthernet0/0/0.91.

The tests I am doing are primarily from a device behind the router but I have also tested from the router itself. 

As you have mentioned the path to Google isn't via AS3223 but it stops working once we raise the session to AS3223 which is the link between the two. Other traffic however continues to work fine such as to Bing. There may be other destinations that don't work too however Google was just an obvious one that was noticed.

Inbound traffic from AS3223 works when we raise the session and if we add a static route for Google via AS3223 it also works.

[Harold Ritter]

Hi @MP13 ,

Could you validate whether the traffic is returning via AS3223. If so, this could be an issue with asymmetric routing. So when you force traffic out AS3223 using a static route, it solves the issue.

Do you have any configuration that could cause that, such as unicast RPF?

Regards,

[MP13]

I've had a look at one of the routes that comes inbound via AS3223 and we appear to send traffic back out via AS174 however that works fine. 

I have checked a few of the interfaces the sessions are built on and they show "IP unicast RPF check is disabled"

I found some notes of the output I made previously where the interface changes which was when checking CEF but it looks like it may be because we learn a more specific route:

sh ipv6 cef 2607:f8b0:4007:816::2004
2607:F8B0::/32
  nexthop FE80::8AA2:5EFF:FE16:887B TenGigabitEthernet0/0/0.91

sh ipv6 cef 2607:f8b0:4007:816::2004
2607:F8B0:4007::/48
  nexthop FE80::211:11FF:FE2A:7ECB TenGigabitEthernet0/0/0.89

[Harold Ritter]

Hi @MP13 ,

> show "IP unicast RPF check is disabled"

This is for ipv4 unicast RPF. Can you check if any of the interfaces have "ipv6 verify source" configured?

Regards,

[MP13]

I've had a check and that command is not set on the interfaces. The only IPv6 command we have on the interface at the moment is the ipv6 address itself.

[MHM Cisco World]

MPLS label: nolabel <<- no labal for this traffic ?? can I know why ? is this pure BGP or VPNv4?

[MP13]

This is pure BGP.

[MHM Cisco World]

I see IPv4 and IPv6 in top of prefix so I think there is issue in Next-hop of prefix 
check ciscolive recommend for case similar to this 
Dissecting NH for IPv6
IPv4 Peering for IPv6 AFI
• When using IPv4 peering, NH is automatically set to IPv4-mapped IPv6
address ::ffff:X.X.X.X (on IOS and IOS-XE) (NX-OS won’t install the prefix)
• This cannot be used as an IPv6 NH since it is an inaccessible address
• RR cannot advertise this prefix
• Exception: Can be accessible if the session is for 6PE or 6VPE
• Alternate Solution
• Have the advertising BGP speaker set an outbound route-map to set the NH to
a global IPv6 address
• Have the receiving BGP speaker set an inbound route-map to set the NH to
a global IPv6 address

[Harold Ritter]

Hi @MHM Cisco World ,

This is not ipv6 prefixes over bgp ipv4 transport. This is  ipv6 prefixes over bgp ipv6 transport.

Regards,

[MHM Cisco World]

I see IPv4 appear in show ipv6 

For example 

(XXXXXX.225.230)

So this mess in next-hop between ipv4 and ipv6' 

Let him try use set global ipv6 inbound and check it effect.

[Harold Ritter]

Hi @MHM Cisco World ,

This is the router id. In ipv6, the router id is still a 32 bit value.

Regards,

[MHM Cisco World]

Are your router is asr 9k IOS-XR ??

[MP13]

ASR 1K IOS-XE