Showing results for 
Search instead for 
Did you mean: 

ERPSAN Traffic and Port channel load balancing


Hi all,

We have ERSPAN configured on Nexus1000v to monitor the web server traffic.We cant see any TCP out of order traffic on the web server. But On the destination of ERSPAN, we can see a lot of "TCP Out of order" traffic (about 12% -15%). According to my knoweldege, TCP of out of order is due to multipaths between source and destination.

So we went to check our network infrastructure.

Our network topology is as below:

ERSPAN Source ---UCS-----Nexus 7k pair ----FW (Active-passive cluster)----Nexus 7k pair---UCS-----ERSPAN Destination.  

Nexus 7k is default gw for both ERSPAN source and destination. ERSPAN and ERSPAN destination in different layer 3 network (VRF), which can talk to each other only through upstream firewall cluster. ERSPAN Source and ERSPAN destination are running on the same UCS physical infrastrcture. They are both VM.

There are  vPC port-channel between UCS FI pair and uplink 7k switches. It is layer 3 connection between 7k pair and firewall cluster.

Below is our finding:

(1) Nexus1000v, we use MAC pinning as load sharing method, which means single path for individual ERSPAN between ERSPAN source and UCS FI.

(2) On UCS,

# show port-channel load-balance

Port Channel Load-Balancing Configuration:

System: source-dest ip

Port Channel Load-Balancing Addresses Used Per-Protocol:

Non-IP: src-dst mac

IP: source-dest ip source-dest mac

As ERPSAN are IP traffic, "source-dest ip source-dest mac" load balancing is used, which means the traffic always hits the same 7k using the same physical link.

(3) On the Nexus 7k, the load balancing method is as below:

# show port-channel load-balance

Port Channel Load-Balancing Configuration:

System: src-dst ip

Port Channel Load-Balancing Addresses Used Per-Protocol:

Non-IP: src-dst mac

IP: src-dst ip

Same as Step 2 we think that load balancing of vPC will make sure that always use the same path to hit the same firewall (active firewall) port as we have same source and destination IP in single ERSPAN source and dest pair. (e.g. ERSPAN source is and ERSPAN dest is

(4) On the firewall, the load balancing method is per-flow. we checked the session table and only see one path for the GRE traffic from one ERSPAN source to one ERSPAN destination, which means the traffic always go back to same 7k.

From firewall back to ERSPAN destination, it will be single path for one ERSAP source and destination pair as well if my above analysis is right.

My question is: is there any possiblity that UCS or Nexus 7k load balancing method handles ERSPAN GRE traffic differently from normal IP traffic, which creates multiple paths for ERSPAN flow?

Your help is highly appreciated!

2 Replies 2

Steve Fuller

Hi David,

Are you not looking at the load-balancing / forwarding paths of the wrong traffic here?

What you've provided above are details of how the ERSPAN traffic is balanced / routed between the Nexus 1000V and the ERSPAN destination. The TCP Out of Order reported by your capture is not related to this, but rather to the traffic between the web server and the hosts connecting to that web server.

You need to understand where the clients are that connect to the web server and check the paths between those and the web server.


Hi Steve,

We have did the TCPDUMP on our web server. We cant see any TCP out of order of inbound and outbound traffic for web server.

So we suspected that TCP out of order is introduced between ERSPAN source and destination.

Best Regards

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Recognize Your Peers