topic Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection in DevNet Sandbox

XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

mood977 — Tue, 12 Nov 2024 19:39:15 GMT

Hi Guys,

I've lunched the XRd Sandbox which include multiple XRd devices as a docker containers ..

I'm not able to form BGP neighborship, I assume the TCP port 179 is not open ..

Do anyone faced the same issue and how to solve it ?

Noting that I don't have much experience with docker.

Thanks

RP/0/RP0/CPU0:xrd-2#show bgp vpnv4 unicast summary

Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 6 6 6 6 6 0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
100.100.100.108 0 100 0 0 0 0 0 00:00:00 Active

RP/0/RP0/CPU0:xrd-2#sho tcp brief
0x00007fc8b00086b0 0x60000000 0 0 100.100.100.102:32388 100.100.100.107:4189 SYNSENT
0x00007fc8a800d940 0x60000000 0 0 100.100.100.102:24368 100.100.100.107:179 SYNSENT
0x00007fc8c000c740 0x60000000 0 0 100.100.100.102:56718 100.100.100.108:179 SYNSENT

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

bigevilbeard — Wed, 13 Nov 2024 13:48:31 GMT

@mood977 the question would be, is this designed to be open? As much as a great resource the sandbox can be, they are set up/designed with security posture in mind which makes this often not possible to change the set up here.

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

mood977 — Wed, 13 Nov 2024 14:09:43 GMT

Thanks for your response, but the purpose of the lab is to do some segment routing with a sample L3VPN service on top .. and that will not work without BGP.

Moreover, the lab includes an XRd node acts as a PCE and pce peering is not forming as well.

Snap from the lab instructions:

Thanks

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

bigevilbeard — Wed, 13 Nov 2024 14:50:01 GMT

@mood977 right... in which case it is designed for such. I looked at the set up in the sandbox guide, is this all up correctly, I see the compose has https://github.com/CiscoDevNet/XRd-Sandbox/blob/main/docker-compose.xr.yml which should bring the up peering when applied?

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

mood977 — Wed, 13 Nov 2024 15:02:40 GMT

Hi,

Yes all the containers are up with the startup conf following up the instructions guide..

Even I can see that the isis is forming adjacency as expected.

Thanks

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

bigevilbeard — Wed, 13 Nov 2024 16:19:56 GMT

@mood977 on the devbox, try netstat -tuln | grep 179 or docker network inspect <network_name> might show you some info.

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

mood977 — Wed, 13 Nov 2024 18:39:57 GMT

@bigevilbeard I appreciate your valuable time trying to help ..

Unfortunately, the net-tools package is not installed in the VM.

One way to check it yourself is to Lunch the XRd Sandbox, it takes less than 10 min to setup, but it requires AnyConnect to connect to the VPN.

No specific conf is done from my side, just the default conf ..

Again, thank you very much for your kind support .. appreciated!

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

Harold Ritter — Thu, 14 Nov 2024 23:43:34 GMT

Hi @mood977 ,

The issue is that the IPv4 dataplane does not work properly. I identified a potential issue with the XRd sandbox configuration and contacted the sandbox owner for him to fix it. I will let you know once it is fixed.

Regards,

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

bigevilbeard — Fri, 15 Nov 2024 10:39:26 GMT

Amazing @Harold Ritter

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

mood977 — Sat, 16 Nov 2024 08:37:49 GMT

Great @Harold Ritter , waiting for your feedback.

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

Jesus Illescas — Thu, 21 Nov 2024 12:23:16 GMT

Thanks @mood977 for reporting the issue and thanks @Harold Ritter for pinging.

To fix the issue we need to disable the bridge IP tables. This is actually displayed when running the xrd-host script (expand the output summary on https://github.com/CiscoDevNet/XRd-Sandbox?tab=readme-ov-file#host-check ).

For xr-compose to be able to use Docker bridges, bridge IP tables must be disabled. Note that there may be security considerations associated with doing so.
Bridge IP tables can be disabled by setting the kernel parameters net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables to 0. These can be modified by adding 'net.bridge.bridge-nf-call-iptables=0' and 'net.bridge.bridge-nf-call-ip6tables=0' to /etc/sysctl.conf or in a dedicated conf file under /etc/sysctl.d/.

I tested the fix and works. Next is to work with the sandbox team to push the change along with other details found by Harold. Changes happen on Thursday, so expect next week to have it done if no issues happen.

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

Harold Ritter — Thu, 21 Nov 2024 15:15:18 GMT

Thanks @Jesus Illescas for fixing the issue

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

mood977 — Thu, 21 Nov 2024 16:01:59 GMT

@Jesus Illescas wrote:
Thanks @mood977 for reporting the issue and thanks @Harold Ritter for pinging.

To fix the issue we need to disable the bridge IP tables. This is actually displayed when running the xrd-host script (expand the output summary on https://github.com/CiscoDevNet/XRd-Sandbox?tab=readme-ov-file#host-check ).
For xr-compose to be able to use Docker bridges, bridge IP tables must be disabled. Note that there may be security considerations associated with doing so.
Bridge IP tables can be disabled by setting the kernel parameters net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables to 0. These can be modified by adding 'net.bridge.bridge-nf-call-iptables=0' and 'net.bridge.bridge-nf-call-ip6tables=0' to /etc/sysctl.conf or in a dedicated conf file under /etc/sysctl.d/.
I tested the fix and works. Next is to work with the sandbox team to push the change along with other details found by Harold. Changes happen on Thursday, so expect next week to have it done if no issues happen.

Thank @Jesus Illescas & @Harold Ritter

Appreciated!

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

Harold Ritter — Wed, 27 Nov 2024 18:31:11 GMT

Hi @mood977 ,

I just launched the XRd sandbox and the issues have been fixed. I would like to thank @Jesus Illescas for his help to get this fixed.

Regards,

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

Jesus Illescas — Fri, 29 Nov 2024 10:32:07 GMT

Hi, as @Harold Ritter shared, the fix is deployed.

It was curious for me how to fix it. The lines to disable the iptables where already present on the sandbox, but I noticed they were not persistent across reboots. So the changes were not applied when a new reservation was done.

The fix was to load the br_netfilter module, so the iptables were loaded and then disabled.

echo "br_netfilter" | sudo tee /etc/modules-load.d/br_netfilter.conf

This make the changes to iptables persistent. I documented this change on the repo https://github.com/CiscoDevNet/XRd-Sandbox?tab=readme-ov-file#how-the-sandbox-was-built

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

Jesus Illescas — Fri, 29 Nov 2024 10:31:06 GMT

What I found more interesting was to see why ISIS worked while BGP doesn't.

It turns out to be specific to the topology we are using, in this case using docker. When xr-compose is used to load a topology of XRd containers, docker bridge networks are used to interconnect the XRd containers.

By default docker bridge will create iptables that will filter IP packets and to keep containers isolated from one another in different networks. As you may know ISIS doesnt use IP while BGP does, that's why we could see ISIS neighborship coming up while BGP was broken.

When you deploy XRd in other environments like K8s, docker bridge networks are not used, therefore this couldn't have happened. So is specific to scenarios using xr-compose.

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

mood977 — Fri, 29 Nov 2024 17:55:26 GMT

I've launched the sandbox and everything looks good.

Again, @Jesus Illescas, @Harold Ritter, & @bigevilbeard appreciate your efforts guys!

Thanks

Re: XRd Sandbox - BGP Status Active Not Able to Form TCP Connection

Harold Ritter — Fri, 29 Nov 2024 18:03:16 GMT

You are very welcome @mood977 and thanks for the feedback