Nexus 1000v Switch shows vethernet port are blocked in VEM after reboot VSM and VEM together or Deleted and add it back

[hcssupport]

Hi,

We have deployed Nexus 1000v Switch in UCS B series environment recently and Vcenter 5.5 version is used.

when i am adding and bring up Nexus 1k very first time it was up and working fine and no issues even i am able to move VM into that new N1K switch.

But, when VSM and VEM rebooted together then immediatly issue get started.

Issue is 1 :  Portchannels are shows down it is associated with BUG ID : CSCur17687  this Bug shows known affected release are 4.2(1)SV2(2.1) and 4.2(1)SV2(2.2)  but we are using N1K version is 5.2(1)SV3(1.5a). So our version also affected by above in this BUG.

And, i followed the work arround in this BUG by re adding the VLANS but additionally one thing missed in this workaround that i need to re add the allowed vlan in trunk under my uplink port-profile in N1K switch.This workarround brought my port channel for the particular VEM and sameway in VEM CLI shows re added vlans are shown.

But, Still i am facing issue as vethernet port are getting blocked when i am moving any VM into Nexus 1 K switch port-profile and getting below error

2015 Nov 27 18:23:44 CHINX1000vA vim[2871]: %VIM-5-IF_ATTACHED: Interface Vethernet3 is attached to Network Adapter 1 of CHISFTP on port 3 of module 3 with dvport id 64
2015 Nov 27 18:23:44 CHINX1000vA ethpm[2809]: %ETHPORT-5-IF_ADMIN_UP: Interface Vethernet3 is admin up .
2015 Nov 27 18:23:44 CHINX1000vA ethpm[2809]: %ETHPORT-5-IF_DOWN_PORT_PROFILE_NOT_FOUND: Interface Vethernet3 is down (port-profile not found)

but, actullay i could see the configuration which has port profile under this veth interface

Vethernet5
 port-group: 2253_SHRD_MGMT_INSIDE
 system vlans: none
 capability l3control: no
 capability iscsi-multipath: no
 capability vxlan: no
 capability l3-vservice: no
 port-profile role: none
 port-binding: static

Module is online and port-channel is up and VMK interface ( part of Nexus 1k switch ) of particular VEM is reachable from VSM.

But still Vethernet is showing down in VSM and Blocked in VEM and Vcenter server.

So if anyone faced this issue please assist me by giving any tips or work arround even if you know the fix (i hope fix is not available yet from cisco) .

Thanks

Naseer

[Reuben Farrelly]

What 'system vlan' settings do you have?  Have you set the VEM-VSM vlan as a system vlan?

[hcssupport]

Hu Reuben Ferrelly,

Yes, I am using Port-Profile with System vlan assigned and this port-profile is L3 capability and access port for that vlan as well.

Moreover, i am using Control, Managment , Packet vlan are same only and all the VSM, VEM, vCenter are in same subnet only.

Please see the sample configuration below.

port-profile type vethernet CONT_VLAN_1222
  switchport mode access
  switchport access vlan 1222
  no shutdown
  capability l3control
  system vlan 1222
  state enabled
  vmware port-group

[Keny Perez]

Please share the output of the following commands from the VSM:

#show svs domain

#show svs connections

#show log | last 20

-Kenny

[hcssupport]

Hi Keny,

Please find the below

connection SRVN_CLD:
    ip address: 10.165.13.100
    remote port: 80
    protocol: vmware-vim https
    certificate: default
    datacenter name: CHI1-SRVN
    admin:
    max-ports: 12000
    DVS uuid: 27 86 06 50 1c 06 5d 94-03 8d 9f 55 52 05 8f 79
    config status: Enabled
    operational status: Connected
    sync status: Complete
    version: VMware vCenter Server 5.5.0 build-2442329
    vc-uuid: 8B9E32C0-6E57-46E7-A14D-8955E572E44A
    ssl-cert: self-signed or not authenticated

SVS domain config:
  Domain id:    100
  Control vlan:  NA
  Packet vlan:   NA
  L2/L3 Control mode: L3
  Switch guid: cdd86da7-26ce-451e-94f7-9cc2ec6f4fa8
  L3 control interface: mgmt0
  Status: Config push to Management Server successful.
  Control type multicast: No
  L3Sec Status: Enabled

Note: Control VLAN and Packet VLAN are not used in L3 mode

sh logging | last 20
2015 Nov 30 14:09:35 CHINX1000vA %VEM_MGR-SLOT4-5-VEM_SYSLOG_NOTICE: ERSPAN-VEM
: L3Control/ERSPAN: module 4 vmknic removed
2015 Nov 30 14:09:33 CHINX1000vA vim[2866]: %VIM-5-IF_CONFIG_PURGE_AUTO: Configu
ration on interface Vethernet2 has been auto purged (port-profile changed)
2015 Nov 30 14:09:33 CHINX1000vA vim[2866]: %VIM-5-IF_ATTACHED: Interface Vether
net2 is attached to vmk2 on port 1 of module 4 with dvport id 544
2015 Nov 30 14:09:33 CHINX1000vA ethpm[2812]: %ETHPORT-5-IF_ADMIN_UP: Interface
Vethernet2 is admin up .
2015 Nov 30 14:09:33 CHINX1000vA ethpm[2812]: %ETHPORT-5-IF_UP: Interface Vether
net2 is up in mode access
2015 Nov 30 14:12:08 CHINX1000vA vim[2866]: %VIM-5-IF_DETACHED: Interface Vether
net5 is detached
2015 Nov 30 14:12:11 CHINX1000vA vim[2866]: %VIM-5-IF_CONFIG_PURGE_AUTO: Configu
ration on interface Vethernet5 has been auto purged (port-profile changed)
2015 Nov 30 14:12:11 CHINX1000vA vim[2866]: %VIM-5-IF_ATTACHED: Interface Vether
net5 is attached to Network Adapter 1 of CHIMADB on port 2 of module 4 with dvpo
rt id 128
2015 Nov 30 14:12:11 CHINX1000vA ethpm[2812]: %ETHPORT-5-IF_ADMIN_UP: Interface
Vethernet5 is admin up .
2015 Nov 30 14:12:11 CHINX1000vA ethpm[2812]: %ETHPORT-5-IF_DOWN_PORT_PROFILE_NO
T_FOUND: Interface Vethernet5 is down (port-profile not found)
2015 Nov 30 14:16:47 CHINX1000vA vim[2866]: %VIM-5-IF_DETACHED: Interface Vether
net5 is detached
2015 Nov 30 14:16:47 CHINX1000vA vim[2866]: %VIM-5-IF_CONFIG_PURGE_AUTO: Configuration on interface Vethernet5 has been auto purged (port-profile changed)
2015 Nov 30 14:16:47 CHINX1000vA vim[2866]: %VIM-5-IF_ATTACHED: Interface Vethernet5 is attached to Network Adapter 1 of CHIMADB on port 2 of module 4 with dvport id 64
2015 Nov 30 14:16:47 CHINX1000vA ethpm[2812]: %ETHPORT-5-IF_ADMIN_UP: Interface Vethernet5 is admin up .
2015 Nov 30 14:16:47 CHINX1000vA ethpm[2812]: %ETHPORT-5-IF_DOWN_PORT_PROFILE_NOT_FOUND: Interface Vethernet5 is down (port-profile not found)
2015 Nov 30 14:18:07 CHINX1000vA vim[2866]: %VIM-5-IF_ATTACHED: Interface Vethernet3 is attached to Network Adapter 1 of CHIMCCDMWEBB on port 3 of module 4 with dvport
id 128
2015 Nov 30 14:18:07 CHINX1000vA ethpm[2812]: %ETHPORT-5-IF_ADMIN_UP: Interface Vethernet3 is admin up .
2015 Nov 30 14:18:07 CHINX1000vA ethpm[2812]: %ETHPORT-5-IF_DOWN_PORT_PROFILE_NOT_FOUND: Interface Vethernet3 is down (port-profile not found)

2015 Dec  1 06:58:13 CHINX1000vA vim[2866]: %VIM-5-IF_DETACHED: Interface Vethernet3 is detached
2015 Dec  1 07:01:35 CHINX1000vA vim[2866]: %VIM-5-IF_DELETE_AUTO: Interface Vethernet3 has been auto deleted

[Keny Perez]

I recommend to open a TAC case

-Kenny

[hcssupport]

Hi,

I have opened the Cisco TAC 2 weeks ago and and they found new BUG (internally created but not visible to customer at the moment) and provided workaround as below.

execute the below commands in VEM (ESXI HOST CLI) module

#hotswap.sh -u

the above command will unload the vem functions from that blade (host) and then execute below

#hotswap.sh -l

This command will load the function again in VEM and it will resolve the issue.

but Still FIX is not provided by cisco.

[Keny Perez]

thanks for the feedback.. you have the BUG ID by any chance?

-Kenny

[hcssupport]

Hi, 

Yes,please find this below bug id:

CSCux46100