Why Fabric Interconnects cannot be connected to both MDS?

[sunning_pro]

Dear all,

1. Why the white paper below states that if FI are connected to both MDS, then it will create a single point of failure?

http://www.cisco.com/c/en/us/products/collateral/storage-networking/mds-9500-series-multilayer-directors/white_paper_c11_586100.html

This picture below is being used to depict the scenario for question 2 to 4.

2. And why the best practice of SAN design is to create SAN A and SAN B? Where when the Switch 1 failed, then it will use the SAN B. And at this case, can user configure to connect to the storage using SAN B first, instead of using SAN A first? And by default, does the behavior of the SAN like that, what I mean is it will check when SAN A failed, then it will use SAN B?

3. And, why there are no connections between the left most server and the switch 2? Can't they design that when Switch 1 Failed, then it will use the other link first to the Switch 2, by linking the left most server to switch 2? Then when the core SAN Switch A down, it will use the SAN B connection.

4. Can the server multipath to different SAN? In this case, SAN A and B.

Thanks,

Osbin

[CSCO12029650 · Accepted solution]

Hi Osbin,

 Why the white paper below states that if FI are connected to both MDS, then it will create a single point of failure?

FC protocol build a fabric, using FSPF protocol (like OSPF) so MDS`s, FI (switching mode) Nexus 5k, 7k (storage VDC) form a single fabric using interswitch links.

If some problem is occur in a single device whole fabric get problem (zoning, FSPF, FCNS)

if you connect fabric A and fabric B (connect one FI to switch 1 and switch 2) this means it is no fabric A and B - it become a single fabric (single fabric = single point of failure). Thats why we use two FI, connected to two independent fabrics (isolation and redundancy)

Technically you can avoid this problem using VSAN, but if some problem occure in a NX-OS or SAN-OS you can loose both fabrics in a single failure (the worst scenario in SAN world)

2. And why the best practice of SAN design is to create SAN A and SAN B? Where when the Switch 1 failed, then it will use the SAN B. And at this case, can user configure to connect to the storage using SAN B first, instead of using SAN A first? And by default, does the behavior of the SAN like that, what I mean is it will check when SAN A failed, then it will use SAN B?

Main point is in answer above, initiator may use both fabrics at the same time (multipath) with special drivers.

It is user configurable to use any fabric (A and B - is just for engineers) first.

It depends on OS and drivers.

3. And, why there are no connections between the left most server and the switch 2?

I believe this picture is schematic diagram, and imagine if you have not enough ports in switch 1

 Can't they design that when Switch 1 Failed, then it will use the other link first to the Switch 2, by linking the left most server to switch 2?

Remember the FC fabric idea. if you loose switch 1 it will affect on every single switch in fabric A, and it is much more quick to use stable fabric B instead of converging fabric A. 

Then when the core SAN Switch A down, it will use the SAN B connection.

4. Can the server multipath to different SAN? In this case, SAN A and B.

Yes

Best regards,

Alex

[sunning_pro]

Dear Alex,

Thanks for your answer. But,

Do u have any references to prove that

"If some problem is occur in a single device whole fabric get problem (zoning, FSPF, FCNS)"?

If that's true, then I think now I know why we have SAN A and SAN B here.

Because if we talk about VLAN, people are using VLAN to reduce broadcast domain.

And I don't know why people are using two SANs, SAN A and SAN B, or they use VSAN.

[CSCO12029650]

Hi, i found some documents to read

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/san_switching/b_Cisco_Nexus_5000_Series_NX-OS_SAN_Switching_Configuration_Guide/configuring_and_managing_zones.html

"All other switches in the fabric receive the active zone set so they can enforce zoning in their respective switches." 

"All switches in the Cisco Nexus 5000 Series distribute active zone sets when new E port links come up or when a new zone set is activated in a VSAN. The zone set distribution takes effect while sending merge requests to the adjacent switch or while activating a zone set.", not only N5K family, MDS, N7K and so on

Zoneset is a setting of fabric, not an individual switch, for example if you mistype some zone or zoneset, it will have disruptive effect not only on this particular switch.

FSPF and FCNS also operates IN a VSAN but usually we do not need to touch it, so different VSANs totally independent: each VSAN have it`s own FC services (zonning, etc)

Idea to isolate fabric A and B by physical devices, not by VSAN is: if you have some bug in OS or you have to reboot the switch (whole device, not VSAN) it will cause to rebuild both fabrics, and  if you have separate devices for san A and san B only one fabric will go down (maybe temporarily, depends on problem/situation)

Because if we talk about VLAN, people are using VLAN to reduce broadcast domain.

One more reason to use VSAN - to reduce fabric size and complexity - similar to reduce broadcast domain in LAN world, main difference is in SAN world there is no inter VSAN routing (only devices in a single VSAN can communicate )

Technically we have IVR (inter vsan routing) but it is implemented to isolate FSPF on a non- stable links, for example DCI is best place for IVR

--

Best regards,

Alex