VxLAN Slow Performance when routing

LewisD1 — Fri, 10 Feb 2023 17:51:20 GMT

I have encountered a strange issue which i'm hoping someone can help me figure out.

We have Cisco Nexus 9K switches deployed with VXLAN and MP-BGP EVPN. We have a few VRFs configured on the switches as such we have a few L3VNIs.

When we are routing traffic between networks using the Anycast Gateways on the cisco Nexus we are getting terrible performance.

For these iPerf Tests i have deployed iPerf into the guestshell but the issues are there even when not running in the guestshell

iPerf from Host to Host using just L2 - Gbits/Sec

Client connecting to 10.255.3.12, TCP port 5001
TCP window size: 3.84 MByte (default)
------------------------------------------------------------
[ 3] local 10.255.3.11 port 38534 connected with 10.255.3.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 28.5 GBytes 24.5 Gbits/sec

iPerf from Host to Switch SVI - Kbits/sec

Client connecting to 10.255.3.1, TCP port 5001
TCP window size: 390 KByte (default)
------------------------------------------------------------
[ 3] local 10.255.3.11 port 42228 connected with 10.255.3.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-11.2 sec 896 KBytes 654 Kbits/sec

iPerf Test from Switch SVI to Host - Mbits/Sec

------------------------------------------------------------
Client connecting to 10.255.3.11, TCP port 5001
TCP window size: 520 KByte (default)
------------------------------------------------------------
[ 3] local 10.255.3.1 port 18524 connected with 10.255.3.11 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 116 MBytes 97.2 Mbits/sec

iPerf from Host to Host via L3

I did this test but dont have a grab of it but again around 300Kbit/Sec

Can anyone point me in the right direction of where to look.

Re: VxLAN Slow Performance when routing

Hanfeng Cai — Thu, 02 Mar 2023 08:56:56 GMT

Basically, switch is doing HW forwarding, I don't believe you hit such basic performance issue. You can take a look the physical path of your testing flow and figure out which hop triggers this problem. In case of any low speed issue, I always suspect packets are punt to switch's CPU.

Re: VxLAN Slow Performance when routing

LewisD1 — Thu, 02 Mar 2023 08:59:09 GMT

Forgot to update the ticket. The issue was IP redirects which was hitting the CPU like you said. Once I enabled no IP redirects the issue went away.

Thanks

Re: VxLAN Slow Performance when routing

Hanfeng Cai — Thu, 02 Mar 2023 09:14:13 GMT

That's cool.

topic Re: VxLAN Slow Performance when routing in Data Center Switches

VxLAN Slow Performance when routing

Re: VxLAN Slow Performance when routing

Re: VxLAN Slow Performance when routing

Re: VxLAN Slow Performance when routing