Hi all,
We have ERSPAN configured on Nexus1000v to monitor the web server traffic.We cant see any TCP out of order traffic on the web server. But On the destination of ERSPAN, we can see a lot of "TCP Out of order" traffic (about 12% -15%). According to my knoweldege, TCP of out of order is due to multipaths between source and destination.
So we went to check our network infrastructure.
Our network topology is as below:
ERSPAN Source ---UCS-----Nexus 7k pair ----FW (Active-passive cluster)----Nexus 7k pair---UCS-----ERSPAN Destination.
Nexus 7k is default gw for both ERSPAN source and destination. ERSPAN and ERSPAN destination in different layer 3 network (VRF), which can talk to each other only through upstream firewall cluster. ERSPAN Source and ERSPAN destination are running on the same UCS physical infrastrcture. They are both VM.
There are vPC port-channel between UCS FI pair and uplink 7k switches. It is layer 3 connection between 7k pair and firewall cluster.
Below is our finding:
(1) Nexus1000v, we use MAC pinning as load sharing method, which means single path for individual ERSPAN between ERSPAN source and UCS FI.
(2) On UCS,
# show port-channel load-balance
Port Channel Load-Balancing Configuration:
System: source-dest ip
Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: src-dst mac
IP: source-dest ip source-dest mac
As ERPSAN are IP traffic, "source-dest ip source-dest mac" load balancing is used, which means the traffic always hits the same 7k using the same physical link.
(3) On the Nexus 7k, the load balancing method is as below:
# show port-channel load-balance
Port Channel Load-Balancing Configuration:
System: src-dst ip
Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: src-dst mac
IP: src-dst ip
Same as Step 2 we think that load balancing of vPC will make sure that always use the same path to hit the same firewall (active firewall) port as we have same source and destination IP in single ERSPAN source and dest pair. (e.g. ERSPAN source is 10.1.1.1 and ERSPAN dest is 10.2.1.1)
(4) On the firewall, the load balancing method is per-flow. we checked the session table and only see one path for the GRE traffic from one ERSPAN source to one ERSPAN destination, which means the traffic always go back to same 7k.
From firewall back to ERSPAN destination, it will be single path for one ERSAP source and destination pair as well if my above analysis is right.
My question is: is there any possiblity that UCS or Nexus 7k load balancing method handles ERSPAN GRE traffic differently from normal IP traffic, which creates multiple paths for ERSPAN flow?
Your help is highly appreciated!
