topic HiCan you please post the in Storage Networking

Issue related to NPV / MDS 9148 / MDS 9222i

Surya ARBY — Fri, 03 Jul 2015 10:51:52 GMT

Hi all.

I'm fighting with an issue related to NPV I can't understand / fix since yesterday (it follows my previous message - this one is a little bit longer). I have one MDS 9222i and 1 9148 both running 6.2.5. The 9222i is the core switch and the 9148 is the access switch running NPV (obviously).

I have two parallel non trunking NP uplinks, after building the config everything works fine (hosts are logged through both uplinks, disks are seen...)

Then I save config and reload the 9222i, and problems arise (I tried by reloading only the 9148 several times, in this case the problem never appears).

After the reload both NP uplinks stay down showing

fc1/xx is down (Link failure or not-connected)

As you can see on the 9148 side NP uplinks are stuck in the "pre initialized" state.

MDS-1# show npv status

npiv is disabled

disruptive load balancing is disabled

External Interfaces:
====================
Interface: fc1/11, State: Pre-Initialized
Interface: fc1/12, State: Pre-Initialized

Number of External Interfaces: 2

Server Interfaces:
==================
Interface: fc1/1, VSAN: 100, State: Waiting for External Interface

Number of Server Interfaces: 1

After issuing a "shut / no shut" sequence on the 9148, one of the NP uplink comes up, the other stays "Pre-Initialized". I can issue 100 times "shut / no shut" one of the uplink comes up, the other stays down / pre initialized.

Then I reload the 9148 (2 times) , nothing changes.

Then I reload the 9222i. Both links are down, shut/no shut, one link comes up. Then I reload the 9222i again. Sometimes one of the uplink comes up automatically. I can't get 2 NP uplinks to be up simultaneously (remind that when I built the config the first time both NP uplinks were up and could log into the fabric)

Regarding the uplink never coming up, here is what I get in the output of

show npv internal events | i fc1/11 prev 3 next 3

33) Event:E_DEBUG, length:262, at 979998 usecs after Thu Jul 2 22:38:46 2015
[538976288] E(100,fc1/11) npivp_get_ext_intf_fsm(1272): Is Core VF Capable: FALSE, Is Phy Login Done: FALSE, Is Port Channel: FALSE RID: { Type: Ext-If(0)
, IfIndex: fc1/11(0x100a000), UCD: 0, VSAN: 0, PWWN: 00:00:00:00:00:00:00:00 }, State NPIVP_EXT_IF_ST_PREINIT_DONE

--

56) Event:E_DEBUG, length:262, at 982963 usecs after Thu Jul 2 22:38:36 2015
[538976288] E(100,fc1/11) npivp_get_ext_intf_fsm(1272): Is Core VF Capable: FALSE, Is Phy Login Done: FALSE, Is Port Channel: FALSE RID: { Type: Ext-If(0)
, IfIndex: fc1/11(0x100a000), UCD: 0, VSAN: 0, PWWN: 00:00:00:00:00:00:00:00 }, State NPIVP_EXT_IF_ST_PREINIT_DONE

--

90) Event:E_DEBUG, length:84, at 26052 usecs after Thu Jul 2 22:38:34 2015
[538976288] E(100,fc1/11) npivp_ext_if_fsm_store_state(275): updating Ext If VSAN PSS Info

91) Event:E_DEBUG, length:254, at 26003 usecs after Thu Jul 2 22:38:34 2015
[538976288] E(100,fc1/11) npivp_get_ext_intf_fsm(1272): Is Core VF Capable: FALSE, Is Phy Login Done: FALSE, Is Port Channel: FALSE RID: { Type: Ext-If(0)
, IfIndex: fc1/11(0x100a000), UCD: 0, VSAN: 0, PWWN: 00:00:00:00:00:00:00:00 }, State NPIVP_EXT_IF_ST_DOWN

92) Event:E_DEBUG, length:254, at 25959 usecs after Thu Jul 2 22:38:34 2015
[538976288] E(100,fc1/11) npivp_get_ext_intf_fsm(1272): Is Core VF Capable: FALSE, Is Phy Login Done: FALSE, Is Port Channel: FALSE RID: { Type: Ext-If(0)
, IfIndex: fc1/11(0x100a000), UCD: 0, VSAN: 0, PWWN: 00:00:00:00:00:00:00:00 }, State NPIVP_EXT_IF_ST_DOWN

Any idea ?

HiCan you please post the

Walter Dey — Fri, 03 Jul 2015 12:53:58 GMT

Can you please post the following information, when the setup is running ok

show npv flogi-table (MDS 9148)

show npv status (MDS 9148)

Are your server ports and uplink ports on the MDS 9148 in the same vsan ?

show interface fc 1/11 and fc 1/12 on the MDS 9148

Hi.

Surya ARBY — Fri, 03 Jul 2015 15:21:07 GMT

Hi.

Unfortunately I can't get both uplinks up again !! (and I can't erase and reload the 9222i now).

I'm stuck in the following state on the 9148 :

MDS-1(config)# show vsan membership
vsan 1 interfaces:
    fc1/3             fc1/4             fc1/5             fc1/6
    fc1/7             fc1/8             fc1/9             fc1/13
    fc1/14            fc1/15            fc1/16            fc1/17
    fc1/18            fc1/19            fc1/20            fc1/21
    fc1/22            fc1/23            fc1/24            fc1/25
    fc1/26            fc1/27            fc1/28            fc1/29
    fc1/30            fc1/31            fc1/32            fc1/33
    fc1/34            fc1/35            fc1/36            fc1/37
    fc1/38            fc1/39            fc1/40            fc1/41
    fc1/42            fc1/43            fc1/44            fc1/45
    fc1/46            fc1/47            fc1/48

vsan 100 interfaces:
    fc1/1             fc1/2             fc1/10            fc1/11
    fc1/12

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

MDS-1(config)# show run interface fc 1/1, fc 1/2, fc 1/11, fc 1/12

!Command: show running-config interface fc1/1-2, fc1/11-12
!Time: Fri Jul  3 06:47:06 2015

version 6.2(5)

interface fc1/1
  switchport mode F
  port-license acquire
  no shutdown

interface fc1/2
  switchport mode F
  port-license acquire
  no shutdown

interface fc1/11
  switchport mode NP
  port-license acquire
  no shutdown

interface fc1/12
  switchport mode NP
  port-license acquire
  no shutdown

MDS-1(config)# show npv status

npiv is disabled

disruptive load balancing is enabled

External Interfaces:
====================
  Interface: fc1/11, State: Pre-Initialized
  Interface: fc1/12, VSAN:  100, FCID: 0x0a0100, State: Up

  Number of External Interfaces: 2

Server Interfaces:
==================
  Interface:  fc1/1, VSAN:  100, State: Up
  Interface:  fc1/2, VSAN:  100, State: Up

  Number of Server Interfaces: 2

MDS-1(config)# show npv flogi-table
--------------------------------------------------------------------------------
SERVER                                                                  EXTERNAL
INTERFACE VSAN FCID             PORT NAME               NODE NAME       INTERFACE
--------------------------------------------------------------------------------
fc1/1     100  0x0a0102 21:00:00:24:ff:3e:7d:a2 20:00:00:24:ff:3e:7d:a2 fc1/12
fc1/2     100  0x0a0101 21:00:00:24:ff:3e:7d:af 20:00:00:24:ff:3e:7d:af fc1/12

Total number of flogi = 2.



MDS-1(config)# show interface fc 1/11-12
fc1/11 is down (Link failure or not-connected)
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:0b:54:7f:ee:40:b1:50
    Admin port mode is NP, trunk mode is off
    snmp link state traps are enabled
    Port vsan is 100
    Receive data field Size is 2112
    Beacon is turned off
    5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec,0 bytes/sec, 0 frames/sec
      1 frames input,124 bytes
        0 discards,0 errors
        0 invalid CRC/FCS,0 unknown class
        0 too long,0 too short
      1 frames output,124 bytes
        0 discards,0 errors
      0 input OLS,0  LRR,0 NOS,0 loop inits
      142 output OLS,0 LRR, 71 NOS, 0 loop inits
    Last clearing of "show interface" counters  :never

fc1/12 is up
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:0c:54:7f:ee:40:b1:50
    Admin port mode is NP, trunk mode is off
    snmp link state traps are enabled
    Port mode is NP
    Port vsan is 100
    Speed is 2 Gbps
    Rate mode is dedicated
    Transmit B2B Credit is 200
    Receive B2B Credit is 32
    Receive data field Size is 2112
    Beacon is turned off
    5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec,0 bytes/sec, 0 frames/sec
      144 frames input,12692 bytes
        0 discards,0 errors
        0 invalid CRC/FCS,0 unknown class
        0 too long,0 too short
      123 frames output,22076 bytes
        0 discards,0 errors
      1 input OLS,0  LRR,0 NOS,0 loop inits
      4 output OLS,5 LRR, 1 NOS, 0 loop inits
      32 receive B2B credit remaining
      200 transmit B2B credit remaining
      200 low priority transmit B2B credit remaining
    Interface last changed at Fri Jul  3 06:43:25 2015

    Last clearing of "show interface" counters  :never

ok, this driving me crazy.

Surya ARBY — Fri, 03 Jul 2015 15:26:05 GMT

ok, this driving me crazy. Just after my previous message, I could shut interface fc1/12, then shut / no shut interface fc1/11, and no shut interface fc1/12 and both uplinks are now up !!

MDS-1(config-if)# show npv status

npiv is disabled

disruptive load balancing is enabled

External Interfaces:
====================
  Interface: fc1/11, VSAN:  100, FCID: 0x0a0120, State: Up
  Interface: fc1/12, VSAN:  100, FCID: 0x0a0100, State: Up

  Number of External Interfaces: 2

Server Interfaces:
==================
  Interface:  fc1/1, VSAN:  100, State: Up
  Interface:  fc1/2, VSAN:  100, State: Up

  Number of Server Interfaces: 2



MDS-1(config-if)# show npv flogi-table
--------------------------------------------------------------------------------
SERVER                                                                  EXTERNAL
INTERFACE VSAN FCID             PORT NAME               NODE NAME       INTERFAC
E
--------------------------------------------------------------------------------
fc1/1     100  0x0a0102 21:00:00:24:ff:3e:7d:a2 20:00:00:24:ff:3e:7d:a2 fc1/12
fc1/2     100  0x0a0101 21:00:00:24:ff:3e:7d:af 20:00:00:24:ff:3e:7d:af fc1/11

Total number of flogi = 2.

I'm currently reloading the 9148 to see if it's stable or not

MDS-1# show interface fc 1/11

Surya ARBY — Fri, 03 Jul 2015 15:44:55 GMT

MDS-1# show interface fc 1/11-12
fc1/11 is up
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:0b:54:7f:ee:40:b1:50
    Admin port mode is NP, trunk mode is off
    snmp link state traps are enabled
    Port mode is NP
    Port vsan is 100
    Speed is 2 Gbps
    Rate mode is dedicated
    Transmit B2B Credit is 200
    Receive B2B Credit is 32
    Receive data field Size is 2112
    Beacon is turned off
    5 minutes input rate 80 bits/sec,10 bytes/sec, 0 frames/sec
    5 minutes output rate 112 bits/sec,14 bytes/sec, 0 frames/sec
      31 frames input,3044 bytes
        0 discards,0 errors
        0 invalid CRC/FCS,0 unknown class
        0 too long,0 too short
      25 frames output,4196 bytes
        0 discards,0 errors
      0 input OLS,0  LRR,0 NOS,0 loop inits
      1 output OLS,2 LRR, 0 NOS, 0 loop inits
      32 receive B2B credit remaining
      200 transmit B2B credit remaining
      200 low priority transmit B2B credit remaining
    Interface last changed at Fri Jul  3 07:01:59 2015

    Last clearing of "show interface" counters  :never

fc1/12 is up
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:0c:54:7f:ee:40:b1:50
    Admin port mode is NP, trunk mode is off
    snmp link state traps are enabled
    Port mode is NP
    Port vsan is 100
    Speed is 2 Gbps
    Rate mode is dedicated
    Transmit B2B Credit is 200
    Receive B2B Credit is 32
    Receive data field Size is 2112
    Beacon is turned off
    5 minutes input rate 88 bits/sec,11 bytes/sec, 0 frames/sec
    5 minutes output rate 112 bits/sec,14 bytes/sec, 0 frames/sec
      32 frames input,3156 bytes
        0 discards,0 errors
        0 invalid CRC/FCS,0 unknown class
        0 too long,0 too short
      25 frames output,4164 bytes
        0 discards,0 errors
      0 input OLS,0  LRR,0 NOS,0 loop inits
      1 output OLS,1 LRR, 0 NOS, 0 loop inits
      32 receive B2B credit remaining
      200 transmit B2B credit remaining
      200 low priority transmit B2B credit remaining
    Interface last changed at Fri Jul  3 07:02:00 2015

    Last clearing of "show interface" counters  :never

MDS-1#

I've reloaded 3 times the 9148, the situation is stable, then I've reloaded the 9222i and same story again 😞



MDS-1(config-if)# show npv status

npiv is disabled

disruptive load balancing is enabled

External Interfaces:
====================
  Interface: fc1/11, State: Pre-Initialized
  Interface: fc1/12, VSAN:  100, FCID: 0x0a0100, State: Up

  Number of External Interfaces: 2

Server Interfaces:
==================
  Interface:  fc1/1, VSAN:  100, State: Up
  Interface:  fc1/2, VSAN:  100, State: Up

  Number of Server Interfaces: 2

MDS-1(config-if)#

fc1/12 is up and fc1/11 shows "link not connected"

MDS-1(config-if)# show int brief | b fc1/11
fc1/11     100    NP     off     notConnected swl    --    --     --
fc1/12     100    NP     off     up           swl   NP     2      --

Can you recover the situation

Walter Dey — Fri, 03 Jul 2015 15:53:33 GMT

Can you recover the situation by shut / no shut of fc 1/11 on the MDS 9148 ?

I have seen such things long time ago, and it was a software bug.

Can you post the versions ?

From what I see, your configuration is ok !

6.2.5 on both units :(about

Surya ARBY — Fri, 03 Jul 2015 15:55:03 GMT

6.2.5 on both units 😞

about the shut /no shut sequences to make both uplinks up or to change the NP uplink which is operationnal, sometimes it works, sometimes it doesn't, it's not predictable.

Just out of curiosity ! Is it

Walter Dey — Fri, 03 Jul 2015 15:58:58 GMT

Just out of curiosity ! Is it always port fc 1/11 ? and if yes, could you swap the cables and see if the problem moves or not ?

yes it's always the same

Surya ARBY — Fri, 03 Jul 2015 16:04:33 GMT

yes it's always the same interface 🙂 in fact I can't swap the cables (I have a remote access) but I could try on another rack with exactly the same devices / versions / cabling map, and i faced exactly the same behavior with the same port numbers, that's why I've not suspected any hardware issue 🙂

I'll try the same config on a nexus 5k next week.

Hi Walter, if someone still

Surya ARBY — Sun, 26 Jul 2015 15:20:22 GMT

Hi Walter, if someone still follow this thread, or if someone in the future faces the same issue and find this thread with a google search, I could notice (today !) the following :

- I can get back both NP uplink up if I shutdown both my NP uplinks, then if I purge the FCID on the upstream switch (purge fcdomain fcid vsan 100 in exec mode) and then bring my NP uplinks back with a no shut.

I don't know what's really going on there but it seems to fix the stuff (tried twice).

- of course after writing this message it doesn't work anymore huhuhu

/ crazy