More than two vSAN's allowed in UCS. Is it possible?

[andyblada1]

Is it possible to have more than one FC port-channel on the UCS carrying different VSAN.?
Two vHBA’s will be created mapped to each FI for vSphere VMFS datastore & other Two vHBA’s will be created mapped to each Fabric Interconnect for tape connectivity.

Total 4 vHBA's needed, basically traffic segregation.

Current Setup is one FC port-channel with 4 Interface on each Fabric Interconnect, carrying 2 vSAN A & B.
Now, I want to introduce 2 new VSAN C & D in the existing setup but with segregation. FI's are configured in End Host Mode.

[Walter Dey · Accepted solution]

Yes ! Just enable port channel, but no trunking !

I did this for a customer recently

Fabric A:              VSAN 10 and 20

Fabric B:              VSAN 11 and 21

This VSAN’s are defined on UCS FI and MDS..

MDS Fabric A

Vsan 10 and 20

f1/1             f1/2

      +             +

      +             +

      +             +

      +             +

      +             +

      +             +

f1/45         f1/46

      UCS FI A

Vsan 10 and 20

F1/1 is in vsan 10

Fc1/2 is in van 20

F1/45 is in vsan 10

F1/46 is in vsan 20

This Links are not Trunks !

vsan 10 Traffic is separated from vsan 20 Traffic between FI und MDS ! traffic between IOM (Chassis) and FI is shared (with a few exceptions, eg. if you have 8 links).

UCS FI: Disable FC Uplink Trunking

see attachment

example UCS FI: Portchannel with 8 Links

san-port-channel 10 is up

    Hardware is Fibre Channel

    Port WWN is 24:0a:54:7f:ee:e8:7e:80

    Admin port mode is NP, trunk mode is off

    snmp link state traps are enabled

    Port mode is NP

    Port vsan is 10

    Speed is 64 Gbps

    1 minute input rate 721252296 bits/sec, 90156537 bytes/sec, 60661 frames/sec    1 minute output rate 604100064 bits/sec, 75512508 bytes/sec, 48866 frames/sec

      6449386961 frames input, 10902378924244 bytes

        17 discards, 0 errors

        0 CRC,  0 unknown class

        0 too long, 0 too short

      3468122012 frames output, 5407839491736 bytes

        0 discards, 0 errors

      8 input OLS, 8 LRR, 8 NOS, 0 loop inits

      8 output OLS, 1 LRR, 1 NOS, 0 loop inits

    last clearing of "show interface" counters never

    Member[1] : fc1/41

    Member[2] : fc1/42

    Member[3] : fc1/43

    Member[4] : fc1/44

    Member[5] : fc1/45

    Member[6] : fc1/46

    Member[7] : fc1/47

    Member[8] : fc1/48

    Interface last changed at Thu Feb 18 14:34:28 2016

fc1/41 is up

    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)

    Port WWN is 20:29:54:7f:ee:e8:7e:80

    Admin port mode is NP, trunk mode is off

    snmp link state traps are enabled

    Port mode is NP

    Port vsan is 10

    Speed is 8 Gbps

    Transmit B2B Credit is 16

    Receive B2B Credit is 16

    Receive data field Size is 2112

    Beacon is turned off

    Belongs to san-port-channel 10

    1 minute input rate 127360912 bits/sec, 15920114 bytes/sec, 9878 frames/sec

    1 minute output rate 84676760 bits/sec, 10584595 bytes/sec, 6701 frames/sec

      806733704 frames input, 1363656296496 bytes

        0 discards, 0 errors

        0 CRC,  0 unknown class

        0 too long, 0 too short

      433985100 frames output, 676730910820 bytes

        0 discards, 0 errors

      1 input OLS, 1 LRR, 1 NOS, 0 loop inits

      1 output OLS, 0 LRR, 0 NOS, 0 loop inits

    last clearing of "show interface" counters never

      16 receive B2B credit remaining

      16 transmit B2B credit remaining

      0 low priority transmit B2B credit remaining

ch-cucsdc00-2-A(nxos)# sho vsan membership

vsan 1 interfaces:

vsan 10 interfaces:

    fc1/41            fc1/42            fc1/43            fc1/44

    fc1/45            fc1/46            fc1/47            fc1/48

    san-port-channel 10 vfc765            vfc773            vfc797

    vfc813            vfc821            vfc829            vfc837

    vfc845            vfc853            vfc861            vfc869

    vfc933            vfc941            vfc949            vfc957

    vfc965            vfc1009           vfc1017           vfc1025

    vfc1035           vfc1036           vfc1049           vfc1073

    vfc1081           vfc1089           vfc1121           vfc1129

    vfc1137           vfc1145           vfc1161           vfc1169

example MDS: Portchannel with 8 Links

CH-NX50DC00-01# sho interface san-port-channel 10

san-port-channel 10 is up

    Port description is CH-CUCSDC00-02-A.SANPO10

    Hardware is Fibre Channel

    Port WWN is 24:0a:00:2a:6a:2c:42:00

    Admin port mode is F, trunk mode is auto

    snmp link state traps are enabled

    Port mode is F

    Port vsan is 10

    Speed is 64 Gbps

    1 minute input rate 480482248 bits/sec, 60060281 bytes/sec, 40082 frames/sec

    1 minute output rate 927926000 bits/sec, 115990750 bytes/sec, 71866 frames/sec

      2330741202173 frames input, 3836673914918548 bytes

        68 discards, 4 errors

        4 CRC,  0 unknown class

        0 too long, 0 too short

      5332487689774 frames output, 9673051107741132 bytes

        0 discards, 9 errors

      16 input OLS, 15 LRR, 13 NOS, 0 loop inits

      66 output OLS, 43 LRR, 44 NOS, 0 loop inits

    last clearing of "show interface" counters never

    Member[1] : fc1/25

    Member[2] : fc1/26

    Member[3] : fc1/27

    Member[4] : fc1/28

    Member[5] : fc1/29

    Member[6] : fc1/30

    Member[7] : fc1/31

    Member[8] : fc1/32

    Interface last changed at Thu Feb 18 15:34:30 2016

[andyblada1]

Hello Walter,

Thank you for your reply, which clear some doubts.

i have confusion here, only vsan can be assigned at a time to a uplink port channel.Refer attached. how can other vsan can be assigned to same port channel.

In drop down window it will allow to select one vsan.

You are saying only port channel will work. Could you please brief more with screenshot.

Regards

Andy

[Walter Dey]

Hi Andy, sorry our messages crossed !

You can do port channel + trunking between UCS FI and MDS ! however, if you do that, every vsan you create in UCS will automatically included in any FC uplink, resp. port-channel; therefore you cannot separate VSAN's in different PC and or links; and if I understand you, this is what you actually want to achieve.

If you only do port channeling (no trunking), you create e.g. on fabric A 2 pc, one in van 10, resp 20 (same for the port-members); and same applies for fabric B.

Hope this is clear by now ?

Walter.

[andyblada1]

Sorry from side too. Seems we were posting at the same time :).

Yes, most of my doubts are clear now. Just rephrasing below.

If i understood you correctly, then i have to split  my port channels ,

1 port channel into two on each FI's. Will it disturb anything??

My current setup is 4 FC uplinks in one PC with vsan 101 on FI A & similarly vsan 102 on FI-B.

Now, i have to split into two PC's on FI-A with vSAN 101 & 111. Same steps for FI-B. vsan 102 & 112.??

One PC with ports 29 & 30 on FI-A vsan 101.

2nd PC with ports 31 & 32 on FI-A vsan 111.

One PC with ports 29 & 30 on FI-B vsan 102.

2nd PC with ports 31 & 32 on FI-B vsan 112.

Will this fine??

[Walter Dey · Accepted solution]

Perfect !

And the members of the pc are in the same vsan as the pc.

If you have one server which has to access both vsan's per fabric, you need 4 vhba's

vhba0 Fabric A vsan 101

vbha1 Fabric B vsan 102

vhba2 Fabric A vsan 111

vhba3 Fabric B vsan 112

Cheers Walter.

[andyblada1]

Thanks a ton Walter for clarifying my all the doubts!!

Last Thing !!

Will it disturb anything? if i am going to split production uplinks from 4 to 2 or removing uplinks from live PC.

Any precautions??

[Walter Dey]

In principle removing links from a pc, requires a relogin of all the sessions using the removed links, and the pc algorithm has to be update.

Due to the fact that you have 2 fabrics, this shouldn't be an issue.

However, in a productive environment, I would be conservative, and recommend scheduling a maintenance window, which could be short (max. 30 minutes).

[andyblada1]

Hello Walter,

I was wondering that instead of splitting the existing port channels, i can also try to create new port channels with new interfaces on FI & MDS switch and assign new vSAN to it. To do that i need new license on FI and as well as on MDS.

Is that right?

[Walter Dey]

Hi Andy

Yes, true !

Q. what kind of FI model do you have ?

Q.a number of licenses are bundled with the FI hardware !

Q. do you have enough FC ports on the FI ? in case of a 62xx, with unified ports, you might need to configure more FC ports, which is disruptive as well.

Walter.

[andyblada1]

1. using 2 X 6248 UP.

2. No spare license is there. Need to purchase SFP and license per port.

3. Yes, empty ports are there and will configure them as FC ports, in case our customer opt for new purchase.

Just curious to know why this new configuration will disruptive?

One more thing adding here, the new vsan will be created under VSAN cloud or Storage Cloud in UCSM. Actually this new VSAN belongs to backup tape library.

Tape library is connected to MDS and MDS is further connected to FI's

[Walter Dey]

3.) configuring unified ports: Ethernet or native FC ports

Changing the port mode on either module can cause an interruption in data traffic because changes to the fixed module require a reboot of the fabric interconnect and changes on an expansion module require a reboot of that module .

If the Cisco UCS domain has a cluster configuration that is set up for high availability and servers with service profiles that are configured for failover, traffic fails over to the other fabric interconnect and data traffic is not interrupted when the port mode is changed on the fixed module.

------------------------------------------------------------------------------------------------------------------------

If you have a MDS, you need to configure VSAN under the SAN cloud ! Storage cloud applies if you have a FC device which is directly connected to FI.

[andyblada1]

Thank you Walter for your speedy responses since last Saturday :).

They were really helpful.

[andyblada1]

After a lot of search found one solution, by enabling FC Uplink trunking on FI.Will there be any traffic disturbance?

Also, it require some changes on MDS switch like enabling fport-channel-trunk feature & NPIV . Will there be any traffic disturbance?

Some other doubts are there like, what i have to choose from drop down on below screen.?

Will there be any impact on my existing production server.

[Walter Dey]

it seems that you didn't study my below entry !

- on MDS, you have to enable NPIV (this is nondisruptive)

- on MDS you have to enable feature fport-channel-trunking; but only use channel, not trunk ! (this is nondisruptive)

- regarding your screen shot: a interface can only be on one vsan ! and because you are only using port channeling, the pc, as well as all it's members are in this vsan.

- same applies to your vhba in the service profile; it belongs to exactly one vsan !