symmetric load balancing in portchannel

[elahe]

Dear community members,

I am trying to use 9k nexus switches with symmetric load balancing to study traffic pattern. I have the following configuration as you can see in the picture. (The linux devices are just a simple bridge. I wanted to be able to see the traffic there using a simple tcpdump command to check if the traffic is symmetrically routed.)

I have disabled the two switches below not to use the vpc as I got that it seems symmetric load balancing algorithm has an issue. So, I decided to check it on a single switch first.

I used both the ip-l4 and ip load balancing but both have problems in making the traffic symmetric. But, although the traffic is balanced, it is not symmetric and the out band traffic dose not come back from the same port.  Any help to solve it or any suggestion on an alternative way to achieve symmetric load balancing will be appreciated. (it seems that sometimes it act symmetrically and sometimes not.) I also add the results of tcpdump. As you can see the 1.1.1.192 to 9.9.9.98 has gone from linux1 which is responsible for the e1/1 interface of the switches but the packet from 9.9.9.98 to 1.1.1.192 has gone from e1/2! As I'm using a traffic generator (ostinato), I could check some different ips and I saw that the last octet of IP seems to affect the hash algorithm and when the two ending octets have some features, the symmetricity is not applied.

It seems that symmetric load balancing algorithm has a problem and needs some attention. If you have any ideas on how to solve it someway, please guide me.

The configuration files are attached.

[MHM Cisco World]

NSK# show port-channel load-balance <<- share this for both NSK

MHM

[elahe]

for all of the switches:
switch(config)# show port-channel load-balance
System config:
Non-IP: src-dst mac
IP: src-dst ip-l4port rotate 0
Option: Symmetric
Port Channel Load-Balancing Configuration for all modules:
Module 1:
Non-IP: src-dst mac
IP: src-dst ip-l4port rotate 0
Option: Symmetric

(I changed the rotate value from 0 to one to check if it solves the issue. but it still has the same problem. I also tried using src-dst ip too.)

[MHM Cisco World]

I will send you some command check your PM

Thanks 

MHM

[MHM Cisco World]

IP: src-dst ip-l4port rotate 0 <<- use hash include only scr-dst IP no more  
dont use L4 port since PC use for source random port

MHM

[balaji.bandi]

You have configuration MAC based LB if you looking IP based check different methods explained here :

https://community.cisco.com/t5/data-center-and-cloud-knowledge-base/load-balancing-methods-for-port-channels-on-nexus/ta-p/3155042

[elahe]

MAC based LB is applied only for non-IP traffic. As I need SYMMETRIC LB, I need to use ip or ip-l4 load balancing as the guide offers. Am I making a mistake? 

[balaji.bandi]

the configuration applies globally based on the decision, this need to be tested, If this production network make sure do in maintenance window, since this required convergence. (this expect to be applied both the side to achieve best optimal results)

Symmetric Hashing

To be able to effectively monitor traffic on a port channel, it is essential that each interface connected to a port channel receives both forward and reverse traffic flows. Normally, there is no guarantee that the forward and reverse traffic flows will use the same physical interface. However, when you enable symmetric hashing on the port channel, bidirectional traffic is forced to use the same physical interface and each physical interface in the port channel is effectively mapped to a set of flows.

When symmetric hashing is enabled, the parameters used for hashing, such as the source and destination IP address, are normalized before they are entered into the hashing algorithm. This process ensures that when the parameters are reversed (the source on the forward traffic becomes the destination on the reverse traffic), the hash output is the same. Therefore, the same interface is chosen.

Only the following load-balancing algorithms support symmetric hashing:

• src-dst ip

• src-dst ip-l4port

check nexus configuration guide :

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/interfaces/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Interfaces_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Interfaces_Configuration_Guide_chapter_0110.html#...

[elahe]

I'm sorry. I did not understand what you meant. I've already used src-dst ip-l4port with symmetric option enabled; but as you can see in the picture provided earlier, the traffic is not transmitted in a symmetric manner.

[balaji.bandi]

I may looked the image due to same name the topology was missed.

Looking at the Topology, how is the Linux side configured ? your PO22 is not vPC if you are testing vPC here (if the Linux is dual homed with nexus)

how are you configured Linux1 and 2 in same port-channel 22 ? (this does not match your configuration you posted)

what kind of testing you conducting, explain the test and explain what devices are part of the test (you have posted 1 and 2) but your testing doing 3 and 4 as per Po 22 concern. and also is the Gateway IP of that sVI in nexus ?

when you testing capture below output from nexus : (both nexus)

show vpc

show port-channel traffic inter port 22

show interface eth x/x (part of all por 22)

suggest to read vPC best practice and understand - how the vPC works - compare to traditional Ether-channel.

https://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

Note : Virtual you may not get 100% results as expected - this is not the right way to test as i per i know., by the way what code you running on the eve-ng for this nexus image nodes ?

[elahe]

the names of the topology and configurations are the same. 

The linux interfaces are just a simple bridge as I needed somewhere to check the traffic flow and find the problem. consider it as a simple wire. e0 is bridged with e2, and e1 with e3. that is completely transparent to the switches and they're there to let me run a tcpdump and understand which port is chosen by the nxos switches.

switch# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 2
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 2
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Delay-restore Orphan-port status : Timer is off.(timeout = 0s)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po20 up 1

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
22 Po22 up success success 1

33 Po33 up success success 1

Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.

switch# show port-channel traffic interface port-channel 22
NOTE: Clear the port-channel member counters to get accurate statistics

ChanId Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
22 Eth1/2 94.34% 0.00% 4.69% 49.88% 0.0% 0.0%
22 Eth1/1 5.65% 100.00% 95.30% 50.11% 100.00% 100.00%
switch#

switch# show interface ethernet 1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Belongs to Po22
Hardware: 100/1000/10000 Ethernet, address: 5006.0000.0101 (bia 5006.0000.0101
)
Description: ***vPC Member Port***
MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 250/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 19:32:09
Last clearing of "show interface" counters never
9 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 936 bits/sec, 1 packets/sec
30 seconds output rate 120 bits/sec, 0 packets/sec
input rate 936 bps, 1 pps; output rate 120 bps, 0 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 216 bits/sec, 0 packets/sec
300 seconds output rate 64 bits/sec, 0 packets/sec
input rate 216 bps, 0 pps; output rate 64 bps, 0 pps
RX
4696 unicast packets 360648 multicast packets 1 broadcast packets
365345 input packets 24491906 bytes
1 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
5066665645597747 input error 0 short frame 0 overrun 0 underrun 0 ignor
ed
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
0 Stomped CRC
TX
18446619157289223281 unicast packets 17379 multicast packets 6391391470842
01984 broadcast packets
639014230663891028 output packets 18446619157292345277 bytes
18446619157289219324 jumbo packets
8445146983038976 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

switch# show interface ethernet 1/2
Ethernet1/2 is up
admin state is up, Dedicated Interface
Belongs to Po22
Hardware: 100/1000/10000 Ethernet, address: 5006.0000.0102 (bia 5006.0000.0102
)
Description: ***vPC Member Port***
MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 19:32:48
Last clearing of "show interface" counters never
9 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 32 bits/sec, 0 packets/sec
30 seconds output rate 32 bits/sec, 0 packets/sec
input rate 32 bps, 0 pps; output rate 32 bps, 0 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 64 bits/sec, 0 packets/sec
300 seconds output rate 64 bits/sec, 0 packets/sec
input rate 64 bps, 0 pps; output rate 64 bps, 0 pps
RX
78367 unicast packets 17778 multicast packets 0 broadcast packets
96145 input packets 9284391 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
0 Stomped CRC
TX
79305 unicast packets 17298 multicast packets 0 broadcast packets
96603 output packets 8440967 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

ps. I added the configuration files.