Problems with MS NLB over vPC using N5K and 6509E

Justin DeVaughn · ‎03-05-2013

I've attached a visio diagram of the configuration for reference.

I'm having two problems:

1. I cannot ping the VIP 10.10.1.1 on the MS NLB servers from other subnets on their ODD numbered IPs. For instance 192.168.1.1 cannot ping the VIP while 192.168.1.2 can ping the VIP.

2. N5K-1 cannot ping the VIP while N5K-2 can ping the VIP.

I have the requisite ARP 10.10.1.1 (insert multicast mac here) ARPA command on the 6509E

I have the Mac address-table static (insert multicast mac here) vlan 10 interface (list interfaces over to N5K e.g...TenGigabitEthernet1/1)

Every connection is vPC'd in the same configuration as below: 6509E --> N5K's --> ESXi hosts

I'm not sure where to begin troubleshooting! These are CAS servers so they are affecting e-mail. any help is definitely appreciated!

Currently the load balancing is being done with the following command:

port-channel load-balance ethernet source-dest-mac

Below is the configuration for the N5K's in sync mode:

interface port-channel1

description ESX1

switchport mode trunk

vpc 1

switchport trunk native vlan 5

switchport trunk allowed vlan 1,5-6,10-12,57-58

spanning-tree port type edge

interface port-channel2

description ESX2

switchport mode trunk

vpc 2

switchport trunk native vlan 5

switchport trunk allowed vlan 1,5-6,10-12,57-58

spanning-tree port type edge

interface port-channel3

description ESX3

switchport mode trunk

vpc 3

switchport trunk native vlan 5

switchport trunk allowed vlan 1,5-6,10-12,57-58

spanning-tree port type edge

interface Ethernet1/1

description ESX1

switchport mode trunk

switchport trunk native vlan 5

switchport trunk allowed vlan 1,5-6,10-12,57-58

channel-group 1

interface Ethernet1/2

description ESX2

switchport mode trunk

switchport trunk native vlan 5

switchport trunk allowed vlan 1,5-6,10-12,57-58

channel-group 2

interface Ethernet1/3

description ESX3

switchport mode trunk

switchport trunk native vlan 5

switchport trunk allowed vlan 1,5-6,10-12,57-58

channel-group 3

Reza Sharifi · ‎03-06-2013

Is vPC configured correctly between the 5ks?

What is is the output of "sh vpc"

What device is the root bridge?

Justin DeVaughn · ‎03-11-2013

Reza,

Attached are the commands you were looking for.

Interestingly the N5K1 STP roots across the vPC peer link with N5K2

N5K2 STP roots across the PortChannel up to the 6500...

NEXUS5K#1

VLAN0001

Root ID Priority 200

Address 0015.c7f5.bc01

Cost 2

Port 4348 (port-channel253)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0004

Root ID Priority 8192

Address 0015.c7f5.bc04

Cost 2

Port 4348 (port-channel253)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0005

Root ID Priority 8192

Address 0015.c7f5.bc05

Cost 2

Port 4348 (port-channel253)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0006

Root ID Priority 8192

Address 0015.c7f5.bc06

Cost 2

Port 4348 (port-channel253)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0007

Root ID Priority 32768

Address 0015.c7f5.bc07

Cost 2

Port 4348 (port-channel253)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0008

Root ID Priority 32768

Address 0015.c7f5.bc08

Cost 2

Port 4348 (port-channel253)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0010

Root ID Priority 8192

Address 0015.c7f5.bc0a

Cost 2

Port 4348 (port-channel253)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

# sh vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 253

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status: success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : secondary

Number of vPCs configured : 102

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po253 up 1,4-8,10-12,15,20,30,35,40,44,50-58,60,65,70,72,75

,80,85,90,95,100,105,110,115,120,125,130,135,140,1

45,150,155,160,165,170,175,180,185,190,195,200,205

,210,215,220,225,230,235,240,245,250,500

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

100 Po100 up success success -

101 Po101 up success success -

240 Po240 up success success 1,5-6,10-12

,57-58

241 Po241 up success success 1,5-6,10-12

,57-58

242 Po242 up success success 1,5-6,10-12

,57-58

250 Po250 up success success 1,4-8,10-12

,15,20,30,3

5,40,44,50-

58,60,65,70

,72,75,....

NEXUS5K #2

VLAN0001

Root ID Priority 200

Address 0015.c7f5.bc01

Cost 1

Port 4345 (port-channel250)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0004

Root ID Priority 8192

Address 0015.c7f5.bc04

Cost 1

Port 4345 (port-channel250)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0005

Root ID Priority 8192

Address 0015.c7f5.bc05

Cost 1

Port 4345 (port-channel250)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0006

Root ID Priority 8192

Address 0015.c7f5.bc06

Cost 1

Port 4345 (port-channel250)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0007

Root ID Priority 32768

Address 0015.c7f5.bc07

Cost 1

Port 4345 (port-channel250)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0008

Root ID Priority 32768

Address 0015.c7f5.bc08

Cost 1

Port 4345 (port-channel250)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

VLAN0010

Root ID Priority 8192

Address 0015.c7f5.bc0a

Cost 1

Port 4345 (port-channel250)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

# sh vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 253

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status: success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : primary

Number of vPCs configured : 102

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po253 up 1,4-8,10-12,15,20,30,35,40,44,50-58,60,65,70,72,75

,80,85,90,95,100,105,110,115,120,125,130,135,140,1

45,150,155,160,165,170,175,180,185,190,195,200,205

,210,215,220,225,230,235,240,245,250,500

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

100 Po100 up success success -

101 Po101 up success success -

240 Po240 up success success 1,5-6,10-12

,57-58

241 Po241 up success success 1,5-6,10-12

,57-58

242 Po242 up success success 1,5-6,10-12

,57-58

250 Po250 up success success 1,4-8,10-12

,15,20,30,3

5,40,44,50-

58,60,65,70

,72,75,....

zahir_ahmed · ‎08-28-2013

Hi Justin,

Did you ever get this issue sorted out?

I am about to configure the same and just wanted to check the correct way of doing this with N5K and Cisco 6509E as the L3 device doing all the routing..

Any advice will be much appreciated..

Thanks

Zee

Justin DeVaughn · ‎08-28-2013

Unfortunately, I did not. The ultimate problem with how this affected the customer was that half of their VDI environment sourced off servers connected to N5K1 and the other half through N5K2. The VDI desktops that were sourced through N5K1 servers received odd number IP addresses and could not reach the CAS e-mail Virtual IP Address. I ended up pointing DNS to one of the real IP's instead of the VIP for the VDI environment - since it was easy enough to scale and fixed the issue.

TAC and I took packet captures and were able to detect the traffic switching properly out the N5K2 port with the VMWare -> CAS servers on it after hairpinning through the 6509E etherchannel. At that point it was a VMware/Server issue which was resolved by recommending a hardware network load balancer - I don't believe the customer ever implemented the hardware network load balancer (because DNS band-aided it) but it was the course of action their Microsoft Partner recommended as well.

Peter Koltl · ‎09-07-2013

What are the NIC teaming settings on the servers? How is teaming and NLB combined?