Cisco MDS 9513 - Zone changes issue

[desingraja]

Dear team,

We are using Cisco MDS9513 in our environment to connect the  hosts and the Pure storage array.

Whenever we do any zonechanges, its affecting the other hosts paths logging in and out on the storage array.

 Especially with Aix / Solaris hosts which takes much time to recover its paths.

Any help highly appreciated.

[Walter Dey]

Not really ! just read the commands how to set static domain id and persistent FC-ID !

[Walter Dey]

Regarding Unix/solaris,.....

Warning HP-UX and AIX are two operating systems that utilize the FC ID in the device path to the storage. For a switch to always assign the same FC ID to a device, persistent FC IDs and static domain ID must be configured for the VSAN.

If you get a new FC-ID, you will no more see your storage !!

and here the guide to configure static domain id, and persistent FC-ID

https://www.cisco.com/en/US/docs/storage/san_switches/mds9000/sw/san-os/quick/guide/qcg_ids.html

[desingraja]

Yes, i gone through the article.
It says disruptive.

Also the current status of my switch. If i enable, does it affects the whole VSAN ?

EMC9513-B# show fcdomain domain-list vsan 60

Number of domains: 1
Domain ID WWN
--------- -----------------------
0x3c(60) 20:3c:00:0d:ec:a2:8b:81 [Local] [Principal]

[Walter Dey]

Yes and no !

Caution Changing Domain_IDs and therefore FC_IDs for a device is disruptive, as an end device has to relogin to the fabric (FLOGI) to obtain a new FCID. However, making a Domain_ID static without changing its value is not disruptive.

for your case:

Enter configuration commands, one per line. End with CNTL/Z. 
switch(config)# fcdomain domain 60 static vsan 60 

switch(config)# fcdomain fcid persistent vsan 60

[desingraja]

I did checked and found the persistent is already enabled on my MDS VSAN

VSAN 60
The local switch is the Principal Switch.

Fabric A :

VSAN 50
The local switch is the Principal Switch.

Local switch run time information:
        State: Stable
        Local switch WWN:    20:32:00:0d:ec:3c:06:01
        Running fabric name: 20:32:00:0d:ec:3c:06:01
        Running priority: 128
        Current domain ID: 0x32(50)

Local switch configuration information:
        State: Enabled
        FCID persistence: Enabled
        Auto-reconfiguration: Disabled
        Contiguous-allocation: Disabled
        Configured fabric name: 20:01:00:05:30:00:28:df
        Optimize Mode: Enabled (Scale Restart)
        Configured priority: 128
        Configured domain ID: 0x32(50) (static)

Principal switch run time information:
        Running priority: 128

Fabric B:

Local switch run time information:
        State: Stable
        Local switch WWN:    20:3c:00:0d:ec:a2:8b:81
        Running fabric name: 20:3c:00:0d:ec:a2:8b:81
        Running priority: 128
        Current domain ID: 0x3c(60)

Local switch configuration information:
        State: Enabled
        FCID persistence: Enabled
        Auto-reconfiguration: Disabled
        Contiguous-allocation: Disabled
        Configured fabric name: 20:01:00:05:30:00:28:df
        Optimize Mode: Enabled (Scale Restart)
        Configured priority: 128
        Configured domain ID: 0x3c(60) (static)

Principal switch run time information:
        Running priority: 128

[Walter Dey]

Static Domain id and persistend FC-ID is perfect !

If you could reproduce the error (also the non flogi of the storage) and then post error log from MDS and AIX would help alot !

[desingraja]

I did checkd with AIX team and found that the host using Class 3 and Point to point or fabric.

After zoning, we were able to see the Pure volume, Once the AIX admin started installing on VIOS1 then he rebooted and the pure volume not available due to the wwpn not logged into the storage array.

Whenever the AIX admin, does cfmgr we were able to see the wwpns logged into the Cisco MDS and not into the Pure storage.

Attached the Show logging log from Switch 1 and Switch 2.

Switch 1, the host connected in fc 2/25 and fc 2/26

Switch 2, the host connected in fc 2/25 and fc 2/28

Also when we were checking the  AIX events, found the error message : Virtual FC host adapater detected an error

Issue noticed : The Virtual wwpns are logged into the SAN successfully but for some reason, its not logging to the Storage issue. 

[desingraja]

Attaching show logging log

[desingraja]

My issue something looks like this.

http://www-01.ibm.com/support/docview.wss?uid=isg1IZ79845

But AIX admin says these fixes already will be applied as they are using
6100-09-09

[Walter Dey]

I see many interface down / up events for misc. interfaces: fc 1/2, fc 1/46, 47, 48 fc/25, 28.

This can hardly been explained by a bad transceiver, cable, port......

Btw, You mention UCS, but I assume this AIX servers are separate, not part of a UCS blade ?

[Walter Dey]

Sorry but I don't understand what you mean ?

After zoning, we were able to see the Pure volume, Once the AIX admin started installing on VIOS1 then he rebooted and the pure volume not available due to the wwpn not logged into the storage array.

I assume you mean, the AIX doesn't see the storage ?

Can you please post

"show flogi database vsan x" before and after the reboot.

and

"show zoneset active vsan x"

Whenever the AIX admin, does cfmgr we were able to see the wwpns logged into the Cisco MDS and not into the Pure storage.

Do you mean, before cfmgr you don't see any flogi entries ?

[Walter Dey]

IBM or the AIX admins should tell you, why they are flagging this errors ?

https://supportforums.cisco.com/t5/storage-networking/how-to-see-zone-manage-npiv-wwns-with-aix-lpars/td-p/1353241


How to see/zone/manage NPIV WWNs with AIX/LPARs

To view WWPN for NPIV on AIX:

From the HMC:

a) GUI

     Partition -> Properties

b) Command Line

lshwres --rsubtype fc -m --level lpar -r virtualio

From VIOS:

As NPIV is a "pass-through" mechanism in AIX, all you will be able to see after NPIV creation, is whether the Client is Logged-in or not via:

lsmap -all npiv

You may also check the available ports via:

lsnports

From Client LPAR:

lsdev -Cc adapter | grep fc

lscfg -vpl fcs0 | grep Net (to show wwpn)

Beware of Live Partition Mobility rules with NPIV…they are very strict!


https://supportforums.cisco.com/t5/storage-networking/problems-with-aix-hosts/td-p/1015757

Re: Problems with AIX hosts

Just as an FYI, all pSeries servers from IBM use a rebranded Emulex HBA with IBM's special firmware on it.

From my past experience, when you plug in an HBA from an AIX host to a Cisco switch (my experience was solely with the 9500 series switches) and opened the port (either GUI or by doing a NO SHUT via CLI), the server would not do a FLOGI. I always had to ask the AIX system admins to run a CFGMGR on their side to have their HBA's power up and perform the FLOGI. If they did this step before the port was turned on and I turned on the port at a later time, the port wouldn't come up.

Here are the steps I always followed to get an AIX server to perform a FLOGI for a new install:

1.) Make sure the VSAN has persistent FCID's set. Someone stated this earlier but is only required for AIX 5.2 or lower. In AIX 5.3, you can change it so the the FCID wasn't stored in the ODM on AIX.

2.) Assign port to correct VSAN

3.) GUI: Enable port CLI: NO SHUT the port

4.) Ask your AIX system admin to run CFGMGR to have your HBA's turned on and perform a FLOGI.

5.) Make sure port comes online as an F port and not as an FL port. If it's an FL port, the AIX system admin needs to fix this on their end. Do not try and hard set your port on the switch to F as the HBA will never login until the AIX is fixed.

6.) Perform zoning tasks.

7.) Assign disk and ask AIX system admin to run CFGMGR again to perform a PLOGI to see it's disk.

[Walter Dey]

Can you please post the interface counters of the ports connecting the AIX and the Pure storage.

Show interface fc ..

see also http://docwiki.cisco.com/wiki/Cisco_MDS_SanOS_Troubleshooting_Guide_--_Troubleshooting_Ports