I have a client that is experiencing issues with the Nexus 5000 switches and vPC configuration. They have a pair of Dell servers each with dual port Emulex CNA's installed. When the server(s) are single homed to one Nexus 5000, using bandwidth throughput testing tools they see a sustained averaged throughput between the two servers of around the 9-9.5Gbps mark. However, when they connect the server to both Nexus 5000's in a vPC configuration, the server throughput seems to drop down to around the 6-6.5Gbps mark. The next odd thing is if one of the ports participating in the vPC is shutdown (or the interface is disconnected), the throughput almost immediately ramps up to the 9-9.5Gbps mark. Then if the port/interface is re-established, the throughput drops back down to 6-6.5Gbps.
The Emulex software was used to create a static (no LACP) network team with Destination IP load balancing, although for the throughput tests only a connection between 2 servers was used so the traffic should be sent to/from the same interface for every packet.
The relavant parts of the Nexus 5000's configurations is below:
Nexus #1:
version 4.1(3)N1(1)
feature vpc
vpc domain 1
peer-keepalive destination 192.168.77.6
vrf context management
ip route 0.0.0.0/0 192.168.77.1
spanning-tree vlan 1-3967,4048-4093 priority 4096
interface port-channel27
switchport mode trunk
switchport access vlan 400
switchport trunk native vlan 400
vpc 27
spanning-tree link-type point-to-point
spanning-tree port type edge trunk
spanning-tree guard loop
speed 10000
interface port-channel28
switchport mode trunk
switchport access vlan 400
switchport trunk native vlan 400
vpc 28
spanning-tree link-type point-to-point
spanning-tree port type edge trunk
spanning-tree guard loop
speed 10000
interface port-channel100
switchport mode trunk
vpc peer-link
spanning-tree port type network
speed 10000
interface Ethernet1/9
description To Nexus 5020 (vPC Peer Keepalive Link)
switchport mode trunk
interface Ethernet1/27
switchport mode trunk
switchport access vlan 400
switchport trunk native vlan 400
spanning-tree link-type point-to-point
spanning-tree port type edge trunk
spanning-tree guard loop
channel-group 27
interface Ethernet1/28
switchport mode trunk
switchport access vlan 400
switchport trunk native vlan 400
spanning-tree link-type point-to-point
spanning-tree port type edge trunk
spanning-tree guard loop
channel-group 28
interface Ethernet1/33
description To Nexus 5020 (vPC Peer Link)
switchport mode trunk
channel-group 100
interface Ethernet1/34
description To Nexus 5020 (vPC Peer Link)
switchport mode trunk
channel-group 100
interface Ethernet1/35
description To Nexus 5020 (vPC Peer Link)
switchport mode trunk
channel-group 100
interface Ethernet1/36
description To Nexus 5020 (vPC Peer Link)
switchport mode trunk
channel-group 100
interface mgmt0
ip address 192.168.77.5/25
cfs ipv4 distribute
cfs eth distribute
Nexus #2:
version 4.1(3)N1(1)
feature vpc
vpc domain 1
peer-keepalive destination 192.168.77.5
vrf context management
ip route 0.0.0.0/0 192.168.77.1
spanning-tree vlan 1-3967,4048-4093 priority 8192
interface port-channel27
switchport mode trunk
switchport access vlan 400
switchport trunk native vlan 400
vpc 27
spanning-tree link-type point-to-point
spanning-tree port type edge trunk
spanning-tree guard loop
speed 10000
interface port-channel28
switchport mode trunk
switchport access vlan 400
switchport trunk native vlan 400
vpc 28
spanning-tree link-type point-to-point
spanning-tree port type edge trunk
spanning-tree guard loop
speed 10000
interface port-channel100
switchport mode trunk
vpc peer-link
spanning-tree port type network
speed 10000
interface Ethernet1/9
description To Nexus 5020 (vPC Peer Keepalive Link)
switchport mode trunk
interface Ethernet1/27
switchport mode trunk
switchport access vlan 400
switchport trunk native vlan 400
spanning-tree link-type point-to-point
spanning-tree port type edge trunk
spanning-tree guard loop
channel-group 27
interface Ethernet1/28
switchport mode trunk
switchport access vlan 400
switchport trunk native vlan 400
spanning-tree link-type point-to-point
spanning-tree port type edge trunk
spanning-tree guard loop
channel-group 28
interface Ethernet1/33
description To Nexus 5020 (vPC Peer Link)
switchport mode trunk
channel-group 100
interface Ethernet1/34
description To Nexus 5020 (vPC Peer Link)
switchport mode trunk
channel-group 100
interface Ethernet1/35
description To Nexus 5020 (vPC Peer Link)
switchport mode trunk
channel-group 100
interface Ethernet1/36
description To Nexus 5020 (vPC Peer Link)
switchport mode trunk
channel-group 100
interface mgmt0
ip address 192.168.77.6/25
cfs ipv4 distribute
cfs eth distribute
Server #1 is connected to Ports 27 on both Nexus 5000's, and Server #2 is connected to Port's 28 on both Nexus 5000's. We have not, as yet, tested this all using LACP for negotiation of the virtual Port-Channels.
Can anyone suggest any ideas on what may be wrong or what we should look at? Is the vPC configuration all correct?
Thanks in advance.
