Possible ISL missconfiguration brought path down on another switch

eric.krejci · ‎10-21-2011

Hello,

I'm facing a behavior that is absoluty insane.

quick intro:

a core edge topology with 2 edge switches, 1 MDS 9134 and 1 Nexus 5548 connected (ISL) to a director MDS 9509. they are all running NXOS 5.x

hosts are connected to the 2 switches and the storage is connected to the director.

for maintenance purpose I had to power off the 9134.

when I powered off the 9134, the connectivity between the hosts connected on the Nexus 5548 and the storage (connected to the Director) had been cut.

I'm a bit confused because, of course, the zonning is correct. zones have WWN of CNA and storage ports WWN.

this behave is like if the path from the 5548 to the storage goes by the 9134.

the only way to bring back connectivity is to power on the 9134.

this might be related to switch priority at the FC domain level? the 9134 has a lower priority than the 5548

I must admit that I'm a bit lost and confused about this situation.

If you have any idea I will be more than pleased you to share it with me :-)

thanks

Eric

maminhas · ‎10-21-2011

Are we using npiv npv mode on MDS and Nexus switches.

eric.krejci · ‎10-22-2011

no I'm not using these modes in my fabric.

maminhas · ‎10-22-2011

if no npiv npv feature was seen while running this command

show run | i npiv

show run | i npv (on both mds and nexus switches)

then you may need to narrow down what is the effective datapath used among the switches.

Pls use the appropiate vsan in the example

e.g

mds1-9513# show fspf internal route vsan 10

FSPF Unicast Routes

---------------------------

VSAN Number Dest Domain Route Cost Next hops

-----------------------------------------------

10 0x02(2) 1000 fc1/14

10 0x03(3) 500 fc1/14

mds1-9513# show fcdomain domain-list vsan 10 | i Local

0x01(1) 20:0a:00:0d:ec:2c:54:c1 [Local]

to track the datapath from one hop to another..best path is chosen based on lower cost if there are multiple links to same destination.

in above example , local switch domain 1 is using same interface 1/14 to reach first switch with domain 2 and second switch with domain 3 when sending traffic in vsan 10

eric.krejci · ‎10-22-2011

thank you for the checks. I will check them in 1 day. I will return with results.

Eric

eric.krejci · ‎10-24-2011

Hi,

for a recap, when the switch21 has been powered off for maintenance, the connectivity between hosts connected on the switch11 and the array connected on the switch01 has been lost. we had to bring back online the switch21 to restore connectivity.

the switch01 and switch21 are running the NXOS 5.0(4c) and the switch11 5.0(3)N1(1c)

here are the info from you checks. I've also added a small schema of my topology:

the infos from

switch01:

switch01# show fspf internal route vsan 2

FSPF Unicast Routes
---------------------------
VSAN Number Dest Domain   Route Cost    Next hops
-----------------------------------------------
           2     0x17(23)          250 port-channel 1
           2    0xa7(167)           62 port-channel 2

switch01# show fcdomain domain-list vsan 2 | include Local
0xe9(233) 20:02:00:0d:ec:35:73:81 [Local] [Principal]

switch01# show fcdomain vsan 2
The local switch is the Principal Switch.

Local switch run time information:
        State: Stable
        Local switch WWN:    20:02:00:0d:ec:35:73:81
        Running fabric name: 20:02:00:0d:ec:35:73:81
        Running priority: 1
        Current domain ID: 0xe9(233)

Local switch configuration information:
        State: Enabled
        FCID persistence: Enabled
        Auto-reconfiguration: Disabled
        Contiguous-allocation: Disabled
        Configured fabric name: 20:01:00:05:30:00:28:df
        Optimize Mode: Disabled
        Configured priority: 1
        Configured domain ID: 0x00(0) (preferred)

Principal switch run time information:
Running priority: 1

Interface               Role          RCF-reject
----------------    -------------    ------------
port-channel 1      Downstream       Disabled
port-channel 2      Downstream       Disabled
----------------    -------------    ------------

switch11:

switch11# show fspf internal route vsan 2

FSPF Unicast Routes
---------------------------
VSAN Number Dest Domain   Route Cost    Next hops
-----------------------------------------------
           2     0x17(23)          312 san-port-channel 1
           2    0xe9(233)           62 san-port-channel 1

switch11# show fcdomain domain-list vsan 2 | include Local

0xa7(167) 20:02:00:05:73:cb:86:81 [Local]

switch11# show fcdomain vsan 2
The local switch is a Subordinated Switch.

Local switch run time information:
        State: Stable
        Local switch WWN:    20:02:00:05:73:cb:86:81
        Running fabric name: 20:02:00:0d:ec:35:73:81
        Running priority: 10
        Current domain ID: 0xa7(167)

Local switch configuration information:
        State: Enabled
        FCID persistence: Enabled
        Auto-reconfiguration: Disabled
        Contiguous-allocation: Disabled
        Configured fabric name: 20:01:00:05:30:00:28:df
        Optimize Mode: Disabled
        Configured priority: 10
        Configured domain ID: 0x00(0) (preferred)

Principal switch run time information:
Running priority: 1

switch21:

switch21# show fspf internal route vsan 2

FSPF Unicast Routes
---------------------------
VSAN Number Dest Domain   Route Cost    Next hops
-----------------------------------------------
           2    0xa7(167)          312 port-channel 1
           2    0xe9(233)          250 port-channel 1

switch21# show fcdomain domain-list vsan 2 | include Local

0x17(23) 20:02:00:0d:ec:ea:a9:41 [Local]

switch21# show fcdomain vsan 2
The local switch is a Subordinated Switch.

Local switch run time information:
        State: Stable
        Local switch WWN:    20:02:00:0d:ec:ea:a9:41
        Running fabric name: 20:02:00:0d:ec:35:73:81
        Running priority: 128
        Current domain ID: 0x17(23)

Local switch configuration information:
        State: Enabled
        FCID persistence: Enabled
        Auto-reconfiguration: Disabled
        Contiguous-allocation: Disabled
        Configured fabric name: 20:01:00:05:30:00:28:df
        Optimize Mode: Disabled
        Configured priority: 128
        Configured domain ID: 0x00(0) (preferred)

Principal switch run time information:
Running priority: 1

I've check the traffic on the ISL between the switch01 and switch21 (the one that was powered off) there are regular traffic, but arround 300 Bytes per seconds, no storage traffic at all.

any clue?

many thanks

Eric

maminhas · ‎10-25-2011

Based on above info and topology..

powering down switch21 should not have any impact..for traffic going between host on switch11 and target on switch01

eric.krejci · ‎10-26-2011

I do absolutely agree with you but it does.

when the switch21 goes down, the connectivity between the switch11 and switch01 is cut.

I will go ahead and open a SR because I'm loosing my mind with this one.

Eric

eric.krejci · ‎11-01-2011

Brilliant,

The first feedback I received from my support is that they never seen such behavior.

I might be lost somewhere in the twilight zone!

Eric

dynamoxxx · ‎11-13-2011

Eric,

did you get anywhere with support ?

@dynamoxxx