07-03-2015 03:51 AM
Hi all.
I'm fighting with an issue related to NPV I can't understand / fix since yesterday (it follows my previous message - this one is a little bit longer). I have one MDS 9222i and 1 9148 both running 6.2.5. The 9222i is the core switch and the 9148 is the access switch running NPV (obviously).
I have two parallel non trunking NP uplinks, after building the config everything works fine (hosts are logged through both uplinks, disks are seen...)
Then I save config and reload the 9222i, and problems arise (I tried by reloading only the 9148 several times, in this case the problem never appears).
After the reload both NP uplinks stay down showing
fc1/xx is down (Link failure or not-connected)
As you can see on the 9148 side NP uplinks are stuck in the "pre initialized" state.
MDS-1# show npv status npiv is disabled disruptive load balancing is disabled External Interfaces: ==================== Interface: fc1/11, State: Pre-Initialized Interface: fc1/12, State: Pre-Initialized Number of External Interfaces: 2 Server Interfaces: ================== Interface: fc1/1, VSAN: 100, State: Waiting for External Interface Number of Server Interfaces: 1
After issuing a "shut / no shut" sequence on the 9148, one of the NP uplink comes up, the other stays "Pre-Initialized". I can issue 100 times "shut / no shut" one of the uplink comes up, the other stays down / pre initialized.
Then I reload the 9148 (2 times) , nothing changes.
Then I reload the 9222i. Both links are down, shut/no shut, one link comes up. Then I reload the 9222i again. Sometimes one of the uplink comes up automatically. I can't get 2 NP uplinks to be up simultaneously (remind that when I built the config the first time both NP uplinks were up and could log into the fabric)
Regarding the uplink never coming up, here is what I get in the output of
show npv internal events | i fc1/11 prev 3 next 3 33) Event:E_DEBUG, length:262, at 979998 usecs after Thu Jul 2 22:38:46 2015 [538976288] E(100,fc1/11) npivp_get_ext_intf_fsm(1272): Is Core VF Capable: FALSE, Is Phy Login Done: FALSE, Is Port Channel: FALSE RID: { Type: Ext-If(0) , IfIndex: fc1/11(0x100a000), UCD: 0, VSAN: 0, PWWN: 00:00:00:00:00:00:00:00 }, State NPIVP_EXT_IF_ST_PREINIT_DONE -- 56) Event:E_DEBUG, length:262, at 982963 usecs after Thu Jul 2 22:38:36 2015 [538976288] E(100,fc1/11) npivp_get_ext_intf_fsm(1272): Is Core VF Capable: FALSE, Is Phy Login Done: FALSE, Is Port Channel: FALSE RID: { Type: Ext-If(0) , IfIndex: fc1/11(0x100a000), UCD: 0, VSAN: 0, PWWN: 00:00:00:00:00:00:00:00 }, State NPIVP_EXT_IF_ST_PREINIT_DONE -- 90) Event:E_DEBUG, length:84, at 26052 usecs after Thu Jul 2 22:38:34 2015 [538976288] E(100,fc1/11) npivp_ext_if_fsm_store_state(275): updating Ext If VSAN PSS Info 91) Event:E_DEBUG, length:254, at 26003 usecs after Thu Jul 2 22:38:34 2015 [538976288] E(100,fc1/11) npivp_get_ext_intf_fsm(1272): Is Core VF Capable: FALSE, Is Phy Login Done: FALSE, Is Port Channel: FALSE RID: { Type: Ext-If(0) , IfIndex: fc1/11(0x100a000), UCD: 0, VSAN: 0, PWWN: 00:00:00:00:00:00:00:00 }, State NPIVP_EXT_IF_ST_DOWN 92) Event:E_DEBUG, length:254, at 25959 usecs after Thu Jul 2 22:38:34 2015 [538976288] E(100,fc1/11) npivp_get_ext_intf_fsm(1272): Is Core VF Capable: FALSE, Is Phy Login Done: FALSE, Is Port Channel: FALSE RID: { Type: Ext-If(0) , IfIndex: fc1/11(0x100a000), UCD: 0, VSAN: 0, PWWN: 00:00:00:00:00:00:00:00 }, State NPIVP_EXT_IF_ST_DOWN
Any idea ?
Solved! Go to Solution.
07-03-2015 08:53 AM
Can you recover the situation by shut / no shut of fc 1/11 on the MDS 9148 ?
I have seen such things long time ago, and it was a software bug.
Can you post the versions ?
From what I see, your configuration is ok !
07-03-2015 05:53 AM
Hi
Can you please post the following information, when the setup is running ok
show npv flogi-table (MDS 9148)
show npv status (MDS 9148)
Are your server ports and uplink ports on the MDS 9148 in the same vsan ?
show interface fc 1/11 and fc 1/12 on the MDS 9148
07-03-2015 08:21 AM
Hi.
Unfortunately I can't get both uplinks up again !! (and I can't erase and reload the 9222i now).
I'm stuck in the following state on the 9148 :
MDS-1(config)# show vsan membership vsan 1 interfaces: fc1/3 fc1/4 fc1/5 fc1/6 fc1/7 fc1/8 fc1/9 fc1/13 fc1/14 fc1/15 fc1/16 fc1/17 fc1/18 fc1/19 fc1/20 fc1/21 fc1/22 fc1/23 fc1/24 fc1/25 fc1/26 fc1/27 fc1/28 fc1/29 fc1/30 fc1/31 fc1/32 fc1/33 fc1/34 fc1/35 fc1/36 fc1/37 fc1/38 fc1/39 fc1/40 fc1/41 fc1/42 fc1/43 fc1/44 fc1/45 fc1/46 fc1/47 fc1/48 vsan 100 interfaces: fc1/1 fc1/2 fc1/10 fc1/11 fc1/12 vsan 4079(evfp_isolated_vsan) interfaces: vsan 4094(isolated_vsan) interfaces:
MDS-1(config)# show run interface fc 1/1, fc 1/2, fc 1/11, fc 1/12 !Command: show running-config interface fc1/1-2, fc1/11-12 !Time: Fri Jul 3 06:47:06 2015 version 6.2(5) interface fc1/1 switchport mode F port-license acquire no shutdown interface fc1/2 switchport mode F port-license acquire no shutdown interface fc1/11 switchport mode NP port-license acquire no shutdown interface fc1/12 switchport mode NP port-license acquire no shutdown
MDS-1(config)# show npv status npiv is disabled disruptive load balancing is enabled External Interfaces: ==================== Interface: fc1/11, State: Pre-Initialized Interface: fc1/12, VSAN: 100, FCID: 0x0a0100, State: Up Number of External Interfaces: 2 Server Interfaces: ================== Interface: fc1/1, VSAN: 100, State: Up Interface: fc1/2, VSAN: 100, State: Up Number of Server Interfaces: 2
MDS-1(config)# show npv flogi-table -------------------------------------------------------------------------------- SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE -------------------------------------------------------------------------------- fc1/1 100 0x0a0102 21:00:00:24:ff:3e:7d:a2 20:00:00:24:ff:3e:7d:a2 fc1/12 fc1/2 100 0x0a0101 21:00:00:24:ff:3e:7d:af 20:00:00:24:ff:3e:7d:af fc1/12 Total number of flogi = 2.
MDS-1(config)# show interface fc 1/11-12 fc1/11 is down (Link failure or not-connected) Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:0b:54:7f:ee:40:b1:50 Admin port mode is NP, trunk mode is off snmp link state traps are enabled Port vsan is 100 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec,0 bytes/sec, 0 frames/sec 1 frames input,124 bytes 0 discards,0 errors 0 invalid CRC/FCS,0 unknown class 0 too long,0 too short 1 frames output,124 bytes 0 discards,0 errors 0 input OLS,0 LRR,0 NOS,0 loop inits 142 output OLS,0 LRR, 71 NOS, 0 loop inits Last clearing of "show interface" counters :never fc1/12 is up Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:0c:54:7f:ee:40:b1:50 Admin port mode is NP, trunk mode is off snmp link state traps are enabled Port mode is NP Port vsan is 100 Speed is 2 Gbps Rate mode is dedicated Transmit B2B Credit is 200 Receive B2B Credit is 32 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec,0 bytes/sec, 0 frames/sec 144 frames input,12692 bytes 0 discards,0 errors 0 invalid CRC/FCS,0 unknown class 0 too long,0 too short 123 frames output,22076 bytes 0 discards,0 errors 1 input OLS,0 LRR,0 NOS,0 loop inits 4 output OLS,5 LRR, 1 NOS, 0 loop inits 32 receive B2B credit remaining 200 transmit B2B credit remaining 200 low priority transmit B2B credit remaining Interface last changed at Fri Jul 3 06:43:25 2015 Last clearing of "show interface" counters :never
07-03-2015 08:26 AM
ok, this driving me crazy. Just after my previous message, I could shut interface fc1/12, then shut / no shut interface fc1/11, and no shut interface fc1/12 and both uplinks are now up !!
MDS-1(config-if)# show npv status npiv is disabled disruptive load balancing is enabled External Interfaces: ==================== Interface: fc1/11, VSAN: 100, FCID: 0x0a0120, State: Up Interface: fc1/12, VSAN: 100, FCID: 0x0a0100, State: Up Number of External Interfaces: 2 Server Interfaces: ================== Interface: fc1/1, VSAN: 100, State: Up Interface: fc1/2, VSAN: 100, State: Up Number of Server Interfaces: 2
MDS-1(config-if)# show npv flogi-table -------------------------------------------------------------------------------- SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFAC E -------------------------------------------------------------------------------- fc1/1 100 0x0a0102 21:00:00:24:ff:3e:7d:a2 20:00:00:24:ff:3e:7d:a2 fc1/12 fc1/2 100 0x0a0101 21:00:00:24:ff:3e:7d:af 20:00:00:24:ff:3e:7d:af fc1/11 Total number of flogi = 2.
I'm currently reloading the 9148 to see if it's stable or not
07-03-2015 08:44 AM
MDS-1# show interface fc 1/11-12 fc1/11 is up Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:0b:54:7f:ee:40:b1:50 Admin port mode is NP, trunk mode is off snmp link state traps are enabled Port mode is NP Port vsan is 100 Speed is 2 Gbps Rate mode is dedicated Transmit B2B Credit is 200 Receive B2B Credit is 32 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 80 bits/sec,10 bytes/sec, 0 frames/sec 5 minutes output rate 112 bits/sec,14 bytes/sec, 0 frames/sec 31 frames input,3044 bytes 0 discards,0 errors 0 invalid CRC/FCS,0 unknown class 0 too long,0 too short 25 frames output,4196 bytes 0 discards,0 errors 0 input OLS,0 LRR,0 NOS,0 loop inits 1 output OLS,2 LRR, 0 NOS, 0 loop inits 32 receive B2B credit remaining 200 transmit B2B credit remaining 200 low priority transmit B2B credit remaining Interface last changed at Fri Jul 3 07:01:59 2015 Last clearing of "show interface" counters :never fc1/12 is up Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:0c:54:7f:ee:40:b1:50 Admin port mode is NP, trunk mode is off snmp link state traps are enabled Port mode is NP Port vsan is 100 Speed is 2 Gbps Rate mode is dedicated Transmit B2B Credit is 200 Receive B2B Credit is 32 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 88 bits/sec,11 bytes/sec, 0 frames/sec 5 minutes output rate 112 bits/sec,14 bytes/sec, 0 frames/sec 32 frames input,3156 bytes 0 discards,0 errors 0 invalid CRC/FCS,0 unknown class 0 too long,0 too short 25 frames output,4164 bytes 0 discards,0 errors 0 input OLS,0 LRR,0 NOS,0 loop inits 1 output OLS,1 LRR, 0 NOS, 0 loop inits 32 receive B2B credit remaining 200 transmit B2B credit remaining 200 low priority transmit B2B credit remaining Interface last changed at Fri Jul 3 07:02:00 2015 Last clearing of "show interface" counters :never MDS-1#
I've reloaded 3 times the 9148, the situation is stable, then I've reloaded the 9222i and same story again :(
MDS-1(config-if)# show npv status npiv is disabled disruptive load balancing is enabled External Interfaces: ==================== Interface: fc1/11, State: Pre-Initialized Interface: fc1/12, VSAN: 100, FCID: 0x0a0100, State: Up Number of External Interfaces: 2 Server Interfaces: ================== Interface: fc1/1, VSAN: 100, State: Up Interface: fc1/2, VSAN: 100, State: Up Number of Server Interfaces: 2 MDS-1(config-if)#
fc1/12 is up and fc1/11 shows "link not connected"
MDS-1(config-if)# show int brief | b fc1/11 fc1/11 100 NP off notConnected swl -- -- -- fc1/12 100 NP off up swl NP 2 --
07-03-2015 08:53 AM
Can you recover the situation by shut / no shut of fc 1/11 on the MDS 9148 ?
I have seen such things long time ago, and it was a software bug.
Can you post the versions ?
From what I see, your configuration is ok !
07-03-2015 08:55 AM
6.2.5 on both units :(
about the shut /no shut sequences to make both uplinks up or to change the NP uplink which is operationnal, sometimes it works, sometimes it doesn't, it's not predictable.
07-03-2015 08:58 AM
Just out of curiosity ! Is it always port fc 1/11 ? and if yes, could you swap the cables and see if the problem moves or not ?
07-03-2015 09:04 AM
yes it's always the same interface :) in fact I can't swap the cables (I have a remote access) but I could try on another rack with exactly the same devices / versions / cabling map, and i faced exactly the same behavior with the same port numbers, that's why I've not suspected any hardware issue :)
I'll try the same config on a nexus 5k next week.
07-26-2015 08:20 AM
Hi Walter, if someone still follow this thread, or if someone in the future faces the same issue and find this thread with a google search, I could notice (today !) the following :
- I can get back both NP uplink up if I shutdown both my NP uplinks, then if I purge the FCID on the upstream switch (purge fcdomain fcid vsan 100 in exec mode) and then bring my NP uplinks back with a no shut.
I don't know what's really going on there but it seems to fix the stuff (tried twice).
- of course after writing this message it doesn't work anymore huhuhu
/ crazy
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: