In case anyone is still interested, I opened a TAC case on this because I could see that the information was all out there, but I wanted to make sure that I didn't miss anything that would bite me. It turnes out that it wasn't that big of a deal if you do it correctly.
To remove a switch that is the principal for a VSAN, as was mention earlier in this thread, you need to give it a lower priority and give the switch that you want to make the principal the highest priority. In our case, the "old" switch had a priority of 1 for 3 of our VSANs, so I changed it to the default of 128 and on the switch that was to be the "new" principal, the priority was set to 1. Assuming that you are in config mode on the switch, the commands from the CLI looked like this:
On the "old" switch:
fcdomain priority 128 vsan xx (xx being the number of the VSAN, this sets the switch priority to the default of 128)
fcdomain restart xx (again, xx being the number of the VSAN, This will non-disruptively restart the vsan on the switch and change the priority. Be aware that if you are using static domain ID's the restart may be disruptive on the switch, check the documention for more on this)
On the "new" switch:
fcdomain priority 1 vsan xx (xx being the number of the VSAN, this sets the switch priority to the highest priority, 1)
fcdomain restart xx (again, xx being the number of the VSAN, This will non-disruptively restart the vsan on the switch and change the priority. Be aware that if you are using static domain ID's the restart may be disruptive on the switch, check the documention for more on this)
Do this for every VSAN that the "old" switch is the principal and also on the "new" switch that is to be the principal. From what I saw in the Domain Manager this will change the priority but won't necessarily change the principal assignment. At this point the "old" switch is ready to be removed from the fabric. Obviously, since this switch was the principal, you'll want to make sure that all of the zoneset information for those zones has been replicated to the other switches in the fabric, particularly to the switch that is to become the new principal. Other than an ISL to the rest of the fabric and the connections to the management ports, there was nothing connected to the "old" switch.
At this point you can shutdown the ISL to the "old" switch from the switch that is on the other end of the ISL. At this point you will see the fabric segment (as viewed from Fabric Manager) At this point, I unplugged the management cables from the "old" switch, since I didn't want it talking to the Fabric Manager anymore. I purged the switch and after about 5 or 10 minutes the segmentation went away and the fabric returned to normal, less the "old" switch. There was no disruption to any of the 240+ hosts that were on the affected VSANs and the switch with the priority of 1 became the principal just like it was supposed to.
I think that once the ISL has been brought down, you want to take the switch off of the network or shut it down so that it can't continue to talk to the FM. If this isn't done, FM will see that there are VSANs on that switch that are disconnected from the rest of the fabric and will show that the fabric is segmented. I went through this procedure successfully on a SUP1 MDS 9509 and a MDS 9120 that were running 3.3.1 code. One thing that did make my heart stop for a second was when I went to look at the zonesets after the segmentation cleared up. The zonesets were there, but the zones were gone! Since my pager wasn't vibrating out of the holster, I reasoned that the zones were still there. A quick look at the configs on the switches confirmed this. What had happened, was that the switch that I removed was the seed switch for FM when it discovered the fabric. To correct this I restarted FM and rediscovered the fabric using the "new" principal switch as the seed switch, and all of the zones were back were they were supposed to be.
Just for your piece of mind (read CYA), I recommend opening a TAC case just to have them look over your configs and make any recommendations. In our case, we had to remove IVR from one of our switches and the engineer pointed out a bug that can occur when removing IVR, and a work around for this bug.
Sorry for the long post, but I hope that this information is helpful to someone else that has to do this.
Jeff Blomendahl
Enterprise Systems Engineer
University of Kansas Medical Center
