Re: Appropriate way to pin to manual sub-group with mac-pinning?

ryan.lambert · ‎03-25-2010

Hi folks,

Working through some specifics on our new 1KV deployment still, and I'm at the point where I have the following scenario:

Two iSCSI VLANs -

93 & 103

Each of these has a single port-profile that my 3 host servers use to create their VMK interfaces. I want to static pin one VLAN to a link, and have the second traverse the second uplink, so I effectively load-balance my iSCSI traffic.

I have 3 port-channels. 1 port-channel per server, 2 member links each:

Port-channel1 = server 1, Gig3/5, Gig3/8

Port-channel2 = server 2, Gig4/5, Gig4/8

Port-channel3 = server 3, Gig5/5, Gig5/8

My question is, can I use sub-group 0 & 1 for each pair of uplinks and specify "pinning id 0" and "pinning id 1" under my common vethernet port-profiles, or do I need to have sub-group 0/1 on po1, sub-group 2/3 on po2, and sub-group 4/5 on po3, and then create separate iSCSI port-profiles for each server that has the correct sub-group pinning ids?

Hope that makes sense. Made my head hurt typing it... just trying to get some clarification on the right way to do this.

Thanks!

ryan.lambert · ‎03-25-2010

As an aside here are my port-profiles:

port-profile type vethernet iSCSI-VLAN93

vmware port-group

switchport mode access

switchport access vlan 93

capability iscsi-multipath

no shutdown

system vlan 93

state enabled

port-profile type vethernet iSCSI-VLAN103

vmware port-group

switchport mode access

switchport access vlan 103

capability iscsi-multipath

no shutdown

system vlan 103

state enabled

nenduri · ‎03-25-2010

Hi Ryan,

The pinning of iscsi multipath capable ports is completely orthogonal to vPC-HM pinning. vPC-HM static pinning ("pinning id <sgid>") will not take effect for iscsi multipath ports.

iscsi multipath capable ports (vmknics) get automatically pinnned to one of the Ethernet ports in round robin. User cannot configure this pinning from CLI.

You can check the pinning information by the following command:

vemcmd show iscsi pinning

You can run the same command from VSM as below:

module vem <mod#> exec vemcmd show iscsi pinning

Since vlan 93 and 103 carried on both uplinks, one get pinned to left and another to right.

HTH.

-Naren

ryan.lambert · ‎03-26-2010

Hi Naren,

I actually saw them getting pinned to the same uplinks (saw dynamic macs upstream on same physical switch port for both VLANs).

Here's my output which seems to confirm that:

N1KV-VSM1# module vem 4 exec vemcmd show iscsi pinning
LTL   Device    IfIndex Pinned_Uplink
   50   vmk1      1b030020        20
   51   vmk2      1b030030        20

Any thoughts? If I can't manually pin them and I end up with this, not sure of how to evenly distribute my iSCSI traffic.

VMK2 = 93

VMK1 = 103

Thanks!

nenduri · ‎03-26-2010

Hi Ryan,

Can you send the output of the following commands?

vemcmd show trunk

vemcmd show port

vemcmd show bd 93

vemcmd show l2 93

vemcmd show dr 93

vemcmd show bd 103

vemcmd show l2 103

vemcmd show dr 103

thanks,

Naren

ryan.lambert · ‎03-30-2010

Hi Naren,

We're in the process of rebuilding some servers, since this was a pre-deployment testing of hardware/software. Once we get the fresh hosts in place and attached to the DVS, I will shoot this info over to you. I should be able to replicate this pretty easily.

Should be in the next day or so, I hope.

ryan.lambert · ‎03-31-2010

Hi Naren!

We see same behavior now that new VSM/VEMs are installed, so it did replicate itself.

N1KV-VSM1# module vem 4 execute vemcmd show iscsi pinning
LTL   Device    IfIndex Pinned_Uplink
   48   vmk0      1b030000        20
   49   vmk1      1b030010        20

Attached the requested output for you (vemcmds.txt)

-Ryan

ryan.lambert · ‎03-31-2010

Actually, this is confusing.

Here's a sh mac-address table dyn off my physical switch interfaces (this is different than before):

dc3# sh mac-address-table dynamic interface eth1/2

VLAN MAC Address Type Age Port

---------+-----------------+-------+---------+------------------------------

92 0050.567e.9488 dynamic 90 Eth1/2

93 0050.5675.61d1 dynamic 190 Eth1/2

95 0002.3d40.5b02 dynamic 0 Eth1/2

dc4# show mac-address-table dynamic interface eth1/2

VLAN MAC Address Type Age Port

---------+-----------------+-------+---------+------------------------------

91 0050.564f.aef1 dynamic 0 Eth1/2

96 0002.3d20.5b02 dynamic 0 Eth1/2

96 0050.567c.6d4b dynamic 20 Eth1/2

103 0050.5676.aec0 dynamic 100 Eth1/2

But... a few minutes later (I didn't drop link or anything on my NIC connected to dc3):

dc4# sh mac-address-table dynamic interface eth1/2
VLAN      MAC Address       Type    Age       Port
---------+-----------------+-------+---------+------------------------------
91        0050.564f.aef1    dynamic 10        Eth1/2
93        0050.5675.61d1    dynamic 80        Eth1/2
96        0002.3d20.5b02    dynamic 10        Eth1/2
103       0050.5676.aec0    dynamic 10        Eth1/2

ryan.lambert · ‎04-01-2010

Opened a TAC case on this as well, and waiting to talk to the Engineer, but still haven't had any luck on this.

Hopefully this is just a matter of me doing something incorrectly...

ryan.lambert · ‎04-01-2010

Well... good news and bad news:

Bad news is according to my TAC engineer and the developer he was working with in the background, this is a bug in the current version SV1(2) and pertains only to iSCSI multipath configured port-profiles. They do not properly pin the VMKs to separate uplinks. I don't have a clean workaround for this in my environment, so I am stuck with all of my iSCSI VMKs traversing a single uplink.

Good news is that it is supposed to be fixed in the next release, tentatively later this month.

I didn't get a bug ID for it, but he may shoot it over in his closing notes.

sekrishn · ‎04-01-2010

Hi Ryan,

Looking at the config, the two vmknics are in two different vlans 93 and 103. The iSCSI multipath auto-pinning works on a per-vlan basis. What this means is that when we pin vmk1 on vlan93, we scan the uplinks and pick the least loaded uplink, say Uplink1. When we pin vmk2, its on a different vlan 103 and the scan may pick Uplink1 again, since it is the least loaded on vlan103.

N1K iSCSI multipathing assumes that the multiple paths to the iSCSI targets are on the same subnet, IOW same vlan. Hence we run the uplink selection on a per-vlan basis. If you move both the vmknics to the same vlan, then you should see they getting pinned to different uplinks.

I also updated the TAC engineer, we can discuss the requirements for the paths on two different vlans.

thx,

- senthil

ryan.lambert · ‎04-01-2010

Hi Senthil,

Thanks for the information.

We were/are (we just can't sustain that configuration long term) going to drop all of these back onto a single VLAN (per request of the TAC engineer) just to confirm pinning behavior was working as expected when all iSCSI is on one VLAN.

Unfortunately, due to the issue in this blog below, we are forced into a 2 VLAN setup:

http://virtualgeek.typepad.com/virtual_geek/2009/08/important-note-for-all-emc-clariion-customers-using-iscsi-and-vsphere.html

ryan.lambert · ‎04-06-2010

Senthil,

We've concluded (as expected) the VMKs do seem to pin properly if within the same VLAN.

Of course, referring to my above post, I suppose I'm caught between a pair of stop signs at this point. EMC makes me do it one way, 1KV wants me to do it another way.

bbaughn1008 · ‎05-12-2010

We have the same issue. We have a Netapp that we're connecting to over iSCSI, and to the EMC directive, the recommended design is 2 IPs on 2 VLANs, so we are stuck apparently unable to pin to different uplinks. Any recommendations how to resolve this??

ryan.lambert · ‎05-12-2010

For us, the saving grace is that EMC is fixing the CLARiiON initiator issue in the next several months (tenatively), and we've not exceeded full capacity on our uplinks.

Once we've rolled out the new flare, we may revert to 1 iSCSI VLAN to get appropriate load distribution, unless this has been persued on the Cisco side as a feature request. Sounds like it isn't just limited to EMC, now.

/curious!