VM Network connectivity loss rebooting FI

wcotton_b2net · ‎06-12-2013

Hi, The issue I’m having I’ve created x2 extra vnics for the ucs blades service profiles to add in a iscsi vm network. Since adding the additonal vnics when I bring the blades and vms back online I’m unable to ping the vms on the vm network if I remove the nics from the vms then re-add the nic the vms can ping but if I reboot one of the FIs I have total packet loss.

Config

X2 Nexus 5ks/x2 6248 FIs/x2 UCS 5108

UCM 2.1a/ESXi 5.1 with the latest Cisco enic drivers

Service Profile vNics/vSwitches

0,1 (vMotion/Mgmt)

2,3 (NFS)

4,5 (iSCSI)

6,7 (VM Network) vlan 1

8,9 (FT)

The VM Network works ok on vNics 4,5 just not on 6,7. The vSwitch config is using the defaults with both nics A/A.

Example Switch Config

ip domain-lookup

class-map type qos class-fcoe

class-map type queuing class-fcoe

match qos-group 1

class-map type queuing class-all-flood

match qos-group 2

class-map type queuing class-ip-multicast

match qos-group 2

class-map type network-qos class-fcoe

match qos-group 1

class-map type network-qos class-all-flood

match qos-group 2

class-map type network-qos class-ip-multicast

match qos-group 2

policy-map type network-qos jumbo

class type network-qos class-fcoe

pause no-drop

mtu 2158

class type network-qos class-default

mtu 9000

multicast-optimize

system qos

service-policy type queuing input fcoe-default-in-policy

service-policy type queuing output fcoe-default-out-policy

service-policy type qos input fcoe-default-in-policy

service-policy type network-qos jumbo

slot 1

port 31-32 type fc

ntp source-interface mgmt0

vrf context management

ip route 0.0.0.0/0 10.100.10.3

vlan 1

vlan 410

name SNAPMIRROR

vlan 420

name NFS

vlan 430

name ISCSI

vlan 440

fcoe vsan 230

name FCOE_B

vlan 450

name FCOE_A

vlan 460

name VMOTION

vlan 470

name FAULT-TOLERANCE

vlan 480

name DMZ

vlan 490

name TEST-VLAN

vlan 999

name 0999_Unused_Ports

spanning-tree port type network default

spanning-tree vlan 1-3967 priority 28672

port-channel load-balance ethernet source-dest-port

vpc domain 2

role priority 7000

system-priority 5000

peer-keepalive destination source

delay restore 150

peer-gateway

port-profile default max-ports 512

vsan database

vsan 230 name "Fabric_B"

device-alias database

device-alias name s2-flr-01_e3b pwwn 50:0a:09:88:98:30:cd:38

device-alias name s2-flr-02_e3b pwwn 50:0a:09:88:88:30:cd:38

device-alias name s2-esx-host01_vHBA_B pwwn 20:00:00:25:b5:00:0b:4f

device-alias name s2-esx-host02_vHBA_B pwwn 20:00:00:25:b5:00:0b:5f

device-alias name s2-esx-host03_vHBA_B pwwn 20:00:00:25:b5:00:0b:2f

device-alias name s2-esx-host04_vHBA_B pwwn 20:00:00:25:b5:00:0b:3f

device-alias name s2-esx-host05_vHBA_B pwwn 20:00:00:25:b5:00:0b:0f

device-alias name s2-esx-host06_vHBA_B pwwn 20:00:00:25:b5:00:0b:1f

device-alias name s2-esx-host07_vHBA_B pwwn 20:00:00:25:b5:00:0b:4e

device-alias commit

fcdomain fcid database

vsan 230 wwn 50:0a:09:82:98:30:cd:38 fcid 0x180000 dynamic

vsan 230 wwn 50:0a:09:88:98:30:cd:38 fcid 0x180020 dynamic

! [s2-flr-01_e3b]

vsan 230 wwn 24:e6:54:7f:ee:ac:75:00 fcid 0x180040 dynamic

vsan 230 wwn 20:00:00:25:b5:00:0b:4f fcid 0x180041 dynamic

! [s2-esx-host01_vHBA_B]

vsan 230 wwn 20:00:00:25:b5:00:0b:5f fcid 0x180042 dynamic

! [s2-esx-host02_vHBA_B]

vsan 230 wwn 20:00:00:25:b5:00:0b:2f fcid 0x180043 dynamic

! [s2-esx-host03_vHBA_B]

vsan 230 wwn 20:00:00:25:b5:00:0b:3f fcid 0x180044 dynamic

! [s2-esx-host04_vHBA_B]

vsan 230 wwn 20:00:00:25:b5:00:0b:0f fcid 0x180045 dynamic

! [s2-esx-host05_vHBA_B]

vsan 230 wwn 20:00:00:25:b5:00:0b:1f fcid 0x180046 dynamic

! [s2-esx-host06_vHBA_B]

vsan 230 wwn 20:00:00:25:b5:00:0b:4e fcid 0x180047 dynamic

! [s2-esx-host07_vHBA_B]

vsan 230 wwn 50:0a:09:82:88:30:cd:38 fcid 0x180001 dynamic

vsan 230 wwn 50:0a:09:88:88:30:cd:38 fcid 0x180002 dynamic

! [s2-flr-02_e3b]

interface san-port-channel 230

channel mode active

switchport trunk allowed vsan 230

switchport trunk mode auto

interface port-channel1

description Peer Link

switchport mode trunk

switchport trunk allowed vlan 1,410,420,430,460,470,480,490

spanning-tree port type network

vpc peer-link

interface port-channel10

description s1-ucs-fi1-A

switchport mode trunk

switchport trunk allowed vlan 1,410,420,430,460,470,480,490

spanning-tree port type edge trunk

vpc 10

interface port-channel11

description s1-ucs-fi1-B

switchport mode trunk

switchport trunk allowed vlan 1,410,420,430,460,470,480,490

spanning-tree port type edge trunk

vpc 11

interface port-channel12

description s2-flr-01

switchport mode trunk

switchport trunk allowed vlan 1,420

spanning-tree port type edge trunk

vpc 12

interface port-channel13

description s2-flr-02

switchport mode trunk

switchport trunk allowed vlan 1,420

spanning-tree port type edge trunk

vpc 13

interface port-channel14

description s2-flr-01

switchport mode trunk

switchport trunk allowed vlan 1,430,440

spanning-tree port type edge trunk

vpc 14

interface port-channel15

description s2-flr-02

switchport mode trunk

switchport trunk allowed vlan 1,430,440

spanning-tree port type edge trunk

vpc 15

interface port-channel100

description 3750X - Core Switch

switchport mode trunk

switchport trunk allowed vlan 1,420,490

spanning-tree port type network

vpc 100

interface vfc102

bind interface port-channel15

switchport trunk allowed vsan 230

no shutdown

interface vfc104

bind interface port-channel14

switchport trunk allowed vsan 230

no shutdown

vsan database

vsan 230 interface vfc102

vsan 230 interface vfc104

vsan 230 interface san-port-channel 230

interface fc1/31

switchport description ucs_cluster_B:fc1/31

switchport trunk mode auto

channel-group 230 force

no shutdown

interface fc1/32

switchport description ucs_cluster_B:fc1/32

switchport trunk mode auto

channel-group 230 force

no shutdown

interface Ethernet1/1

description s2-flr-01-e3a

switchport mode trunk

switchport trunk allowed vlan 1,420

channel-group 12 mode active

interface Ethernet1/2

description s2-flr-01-e3b

switchport mode trunk

switchport trunk allowed vlan 1,430,440

channel-group 14 mode active

interface Ethernet1/3

description s2-flr-02-e3a

switchport mode trunk

switchport trunk allowed vlan 1,420

channel-group 13 mode active

interface Ethernet1/4

description s2-flr-02-e3b

switchport mode trunk

switchport trunk allowed vlan 1,430,440

channel-group 15 mode active

interface Ethernet1/5

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/6

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/7

description Peer Link

switchport mode trunk

switchport trunk allowed vlan 1,410,420,430,460,470,480,490

channel-group 1 mode active

interface Ethernet1/8

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/9

description ho-sw-107-g1/1/2

switchport mode trunk

switchport trunk allowed vlan 1,420,490

spanning-tree port type network

speed 1000

channel-group 100 mode active

interface Ethernet1/10

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/11

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/12

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/13

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/14

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/15

description ho-sw-107-g2/1/2

switchport mode trunk

switchport trunk allowed vlan 1,420,490

spanning-tree port type network

speed 1000

channel-group 100 mode active

interface Ethernet1/16

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/17

description s2-ucs-fi1-A(SSI161204X3)-eth1/18

switchport mode trunk

switchport trunk allowed vlan 1,410,420,430,460,470,480,490

channel-group 10 mode active

interface Ethernet1/18

description s2-ucs-fi1-B(SSI16160EZE)-eth1/18

switchport mode trunk

switchport trunk allowed vlan 1,410,420,430,460,470,480,490

channel-group 11 mode active

interface Ethernet1/19

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/20

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/21

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/22

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/23

description Peer Link

switchport mode trunk

switchport trunk allowed vlan 1,410,420,430,460,470,480,490

channel-group 1 mode active

interface Ethernet1/24

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/25

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/26

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/27

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/28

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/29

description SPARE

shutdown

switchport access vlan 999

interface Ethernet1/30

description SPARE

shutdown

switchport access vlan 999

interface mgmt0

ip address

line console

line vty

boot kickstart bootflash:/n5000-uk9-kickstart.5.2.1.N1.2.bin

boot system bootflash:/n5000-uk9.5.2.1.N1.2.bin

interface fc1/31

interface fc1/32

interface fc1/31

interface fc1/32

!Full Zone Database Section for vsan 230

zone name s2-esx-host01_vHBA_B vsan 230

member pwwn 20:00:00:25:b5:00:0b:4f

! [s2-esx-host01_vHBA_B]

member pwwn 50:0a:09:88:98:30:cd:38

! [s2-flr-01_e3b]

member pwwn 50:0a:09:88:88:30:cd:38

! [s2-flr-02_e3b]

zone name s2-esx-host02_vHBA_B vsan 230

member pwwn 20:00:00:25:b5:00:0b:5f

! [s2-esx-host02_vHBA_B]

member pwwn 50:0a:09:88:98:30:cd:38

! [s2-flr-01_e3b]

member pwwn 50:0a:09:88:88:30:cd:38

! [s2-flr-02_e3b]

zone name s2-esx-host03_vHBA_B vsan 230

member pwwn 20:00:00:25:b5:00:0b:2f

! [s2-esx-host03_vHBA_B]

member pwwn 50:0a:09:88:98:30:cd:38

! [s2-flr-01_e3b]

member pwwn 50:0a:09:88:88:30:cd:38

! [s2-flr-02_e3b]

zone name s2-esx-host04_vHBA_B vsan 230

member pwwn 20:00:00:25:b5:00:0b:3f

! [s2-esx-host04_vHBA_B]

member pwwn 50:0a:09:88:98:30:cd:38

! [s2-flr-01_e3b]

member pwwn 50:0a:09:88:88:30:cd:38

! [s2-flr-02_e3b]

zone name s2-esx-host05_vHBA_B vsan 230

member pwwn 20:00:00:25:b5:00:0b:0f

! [s2-esx-host05_vHBA_B]

member pwwn 50:0a:09:88:98:30:cd:38

! [s2-flr-01_e3b]

member pwwn 50:0a:09:88:88:30:cd:38

! [s2-flr-02_e3b]

zone name s2-esx-host06_vHBA_B vsan 230

member pwwn 20:00:00:25:b5:00:0b:1f

! [s2-esx-host06_vHBA_B]

member pwwn 50:0a:09:88:98:30:cd:38

! [s2-flr-01_e3b]

member pwwn 50:0a:09:88:88:30:cd:38

! [s2-flr-02_e3b]

zone name s2-esx-host07_vHBA_B vsan 230

member pwwn 20:00:00:25:b5:00:0b:4e

! [s2-esx-host07_vHBA_B]

member pwwn 50:0a:09:88:98:30:cd:38

! [s2-flr-01_e3b]

member pwwn 50:0a:09:88:88:30:cd:38

! [s2-flr-02_e3b]

zoneset name FlexPod vsan 230

member s2-esx-host01_vHBA_B

member s2-esx-host02_vHBA_B

member s2-esx-host03_vHBA_B

member s2-esx-host04_vHBA_B

member s2-esx-host05_vHBA_B

member s2-esx-host06_vHBA_B

member s2-esx-host07_vHBA_B

zoneset activate name FlexPod vsan 230

Any help appreciated thanks

Will

jj27 · ‎06-14-2013

What Fabric Interconnect are you rebooting? Have you tested rebooting BOTH and see if the traffic loss only occurs when rebooting a certain one?

What fabric are NICs 6 and 7 tied to, A or B?