ports blocked by admin

[Dragomir]

on my 1000v, when i try to move my control nic to the 1000v dvS, packets are not forwarding and the vem loses connectivity to the vsm.

on the 1000v my uplink is configured as

port-profile type ethernet ucs-n1kv-uplink

  vmware port-group

  switchport mode trunk

  mtu 9000

  switchport trunk allowed vlan 1-3967,4048-4093

  channel-group auto mode on mac-pinning

  no shutdown

  system vlan 50,60,220

  state enabled

on the L3control port profile I have

port-profile type vethernet ucs-n1kv-L3control-vlan60

  capability l3control

  vmware port-group

  switchport mode access

  switchport access vlan 60

  no shutdown

  system vlan 60

  state enabled

ON the UCS side, both nics are configured as trunks and it does includes vlan 60.

any idea why it does not forward on the 1k dvs? ut works just fine on the standard vswitch. all the nics are configured as trunks

[Robert Burns]

Is it possible VLAN 60 is configured as the Native VLAN on your UCS vNICs?  Check the vNIC allowed VLANs (including Native) on the vSwitch uplinks and the DVS uplink - compare the two from a UCS perspective - are they identical?

Likely a native VLAN issue...

Send a screen shot of the vSwitch view from vCenter when the VSM's control NIC is currenlty connected.  I want to see the VLAN tag on the Port Group.

Robert

[Dragomir]

Robert, on the UCS side, native vlan is actaully vlan 50. not 60.

this is the vswitch, the control nic is tagged with vlan 60. On the ucs side, native vlan is  50.

on the same server, vmnic0,1 are on standard switch, set as trunk ports and native vlan 50. It does include vlan 60

on the same server, vmnic4,5 are on the 1k dvs, set as trunk ports (no native vlan set). it does include vlan 60

[lwatta]

Can we get a "show svs domain"???

[Dragomir]

vsm-n1kv# sh svs domain

SVS domain config:

  Domain id:    1000

  Control vlan:  NA

  Packet vlan:   NA

  L2/L3 Control mode: L3

  L3 control interface: control0

  Status: Config push to VC successful.

  Control type multicast: No

[Dragomir]

vsm-n1kv# show module

Mod  Ports  Module-Type                       Model               Status

---  -----  --------------------------------  ------------------  ------------

1    0      Virtual Supervisor Module         Nexus1000V          active *

2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

3    248    Virtual Ethernet Module           NA                  ok

4    248    Virtual Ethernet Module           NA                  ok

Mod  Sw                  Hw

---  ------------------  ------------------------------------------------

1    4.2(1)SV2(1.1a)     0.0

2    4.2(1)SV2(1.1a)     0.0

3    4.2(1)SV2(1.1a)     VMware ESXi 5.1.0 Releasebuild-914609 (3.1)

4    4.2(1)SV2(1.1a)     VMware ESXi 5.1.0 Releasebuild-914609 (3.1)

Mod  MAC-Address(es)                         Serial-Num

---  --------------------------------------  ----------

1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA

2    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA

3    02-00-0c-00-03-00 to 02-00-0c-00-03-80  NA

4    02-00-0c-00-04-00 to 02-00-0c-00-04-80  NA

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

1    10.220.220.230   NA                                    NA

2    10.220.220.230   NA                                    NA

3    10.50.2.30       c6456f3e-2aa1-11e2-1100-00000000002f  xxxxxxxxxxxxxxx

4    10.50.2.31       c6456f3e-2aa1-11e2-1100-00000000000e  xxxxxxxxxxxxxxx

vem communicates with vsm fine on vswitch. once moved to dvs on the 1k I get

vsm-n1kv#

vsm-n1kv# 2013 Feb 11 20:16:42 vsm-n1kv %VEM_MGR-2-VEM_MGR_REMOVE_NO_HB: Removing VEM 3 (heartbeats lost)

2013 Feb 11 20:16:42 vsm-n1kv %VEM_MGR-2-VEM_MGR_REMOVE_NO_HB: Removing VEM 4 (heartbeats lost)

2013 Feb 11 20:16:48 vsm-n1kv %SYSMGR-2-SYNC_FAILURE_MSG_PAYLOAD: vdc 1: Failure from active SUP

2013 Feb 11 20:16:48 vsm-n1kv %KERN-2-SYSTEM_MSG: mts_tcp_send_sync_msg():225 max retry count reached when trying to send msg to standby, (-11) - kernel

2013 Feb 11 20:16:48 vsm-n1kv %KERN-2-SYSTEM_MSG: do_xmit_mtsbuf_sync_msg_tcp(): Sync TCP send failed with error -70 - kernel

2013 Feb 11 20:16:48 vsm-n1kv %KERN-2-SYSTEM_MSG: do_mts_standby_sync: failed to sync message, ha_stage 1, opc 1813, error -70 - kernel

2013 Feb 11 20:16:48 vsm-n1kv %KERN-2-SYSTEM_MSG: node=4 sap=1 desc=MTS Sync Thread, rq=1248 lq=0 pq=0 nq=1 sq=0 buf_in_transit=0, bytes_in_transit=0 - kernel

2013 Feb 11 20:16:53 vsm-n1kv %SYSMGR-2-SYNC_FAILURE_STANDBY_RESET: Failure in syncing messages to standby for vdc 1 causing standby to reset.

2013 Feb 11 20:17:01 vsm-n1kv %PLATFORM-2-MOD_REMOVE: Module 2 removed (Serial number T5056B45E2E)

2013 Feb 11 20:17:01 vsm-n1kv %VEM_MGR-2-MOD_OFFLINE: Module 3 is offline

2013 Feb 11 20:17:01 vsm-n1kv %VEM_MGR-2-MOD_OFFLINE: Module 4 is offline

vsm-n1kv# 2013 Feb 11 20:17:35 vsm-n1kv %PLATFORM-2-MOD_DETECT: Module 2 detected (Serial number :unavailable) Module-Type Virtual Supervisor Module Model :unavailable

[Dragomir]

here is the screenshot on the dvs after I moved the control over to the 1kol nic over

[Robert Burns]

Tony,

While the VSMs are showing blocked, can you go to the Console of the ESX host where the primary VSM resides, and get the output of:

vemcmd show trunk

Also from the VSM can you get "show int veth x" where x is the VSM's Control interface.  If you're not sure what vEth # it is, use "show int virtual" to get the vEth mapping.

Thanks,

Robert

[Dragomir]

~ # vemcmd show trunk

Trunk port 16 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(3972) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1, vlan(50) cbl 1, vlan(60) cbl 1, vlan(220) cbl 1, vlan(10) cbl 1,

Trunk port 21 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(50) cbl 1, vlan(60) cbl 1, vlan(220) cbl 1, vlan(10) cbl 1,

Trunk port 22 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(50) cbl 1, vlan(60) cbl 1, vlan(220) cbl 1, vlan(10) cbl 1,

Trunk port 305 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(50) cbl 1, vlan(60) cbl 1, vlan(220) cbl 1, vlan(10) cbl 1,

~ # vemcmd show trunk

Trunk port 16 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(3972) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1, vlan(50) cbl 1, vlan(60) cbl 1, vlan(220) cbl 1, vlan(10) cbl 1,

Trunk port 21 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(50) cbl 1, vlan(60) cbl 1, vlan(220) cbl 1, vlan(10) cbl 1,

Trunk port 22 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(50) cbl 1, vlan(60) cbl 1, vlan(220) cbl 1, vlan(10) cbl 1,

Trunk port 305 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(50) cbl 1, vlan(60) cbl 1, vlan(220) cbl 1, vlan(10) cbl 1,

-------------------------------------------------------------------------------------------------------------

vsm-n1kv# sh int vethernet1

Vethernet1 is down (nonParticipating)

  Port description is VMware VMkernel, vmk5

  Hardware: Virtual, address: 0050.5662.3168 (bia 0050.5662.3168)

  Owner is VMware VMkernel, adapter is vmk5

  Inactive (last active on module 3)

  VMware DVS port 196

  Port-Profile is ucs-n1kv-L3control-vlan60

  Port mode is access

  5 minute input rate 0 bits/second, 0 packets/second

  5 minute output rate 0 bits/second, 0 packets/second

  Rx

    25542 Input Packets 19766 Unicast Packets

    0 Multicast Packets 5776 Broadcast Packets

    5265560 Bytes

  Tx

    50273 Output Packets 7567 Unicast Packets

    24912 Multicast Packets 17794 Broadcast Packets 42706 Flood Packets

    6691492 Bytes

    0 Input Packet Drops 0 Output Packet Drops

[Robert Burns]

Veth1 is not your VSM's Control Interface.  Look for Adapter1 of your VSM VM.   vEth1 in your output is a VMK interface, likely the host's VMKernel for Mgmt, not the VSMs.

The VEM is programmed correctly.  The CBL states are all FWD.

Can you check your licensing:

show mod

show license brief

show license usage

Can you also try a VSM switchover? "system switchover" in case its a stuck SVS Connection.

Thanks,

Robert

[Dragomir]

I am not seeing anything for

sh int virtual

vsm-n1kv# sh int virtual

-------------------------------------------------------------------------------

Port        Adapter        Owner                    Mod Host

-------------------------------------------------------------------------------

vsm-n1kv# show mod

Mod  Ports  Module-Type                       Model               Status

---  -----  --------------------------------  ------------------  ------------

1    0      Virtual Supervisor Module         Nexus1000V          active *

2    0      Virtual Supervisor Module                             powered-up

Mod  Sw                  Hw

---  ------------------  ------------------------------------------------

1    4.2(1)SV2(1.1a)     0.0

Mod  MAC-Address(es)                         Serial-Num

---  --------------------------------------  ----------

1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

1    10.220.220.230   NA                                    NA

------------------------------------------------------------------------------------------------------

vsm-n1kv# sh mod vem mapping

Mod     Status          UUID                                    License Status

---     -----------     ------------------------------------    --------------

3       absent          c6456f3e-2aa1-11e2-1100-00000000002f    licensed

4       absent          c6456f3e-2aa1-11e2-1100-00000000000e    licensed

vsm-n1kv# sh mod vem missing

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

4    10.50.2.31       c6456f3e-2aa1-11e2-1100-00000000000e  NA

3    10.50.2.30       c6456f3e-2aa1-11e2-1100-00000000002f  NA

----------------------------------------------------------------------------------------------------------

sh license brief is blank

vsm-n1kv# sh license usage

Feature                      Ins  Lic   Status Expiry Date Comments

                                 Count

--------------------------------------------------------------------------------

NEXUS_VSG_SERVICES_PKG        No  512   Unused 05 Apr 2013 -

NEXUS1000V_LAN_SERVICES_PKG   No  512   Unused 05 Apr 2013 -

NEXUS_ASA1000V_SERVICES_PKG   No   16   Unused 05 Apr 2013 -

--------------------------------------------------------------------------------

[Dragomir]

i noticed my control0 interface is down

vsm-n1kv# show system redundancy status

Redundancy role

---------------

      administrative:   primary

         operational:   primary

Redundancy mode

---------------

      administrative:   HA

         operational:   None

This supervisor (sup-1)

-----------------------

    Redundancy state:   Active

    Supervisor state:   Active

      Internal state:   Active with warm standby

Other supervisor (sup-2)

------------------------

    Redundancy state:   Standby

Control connectivity is DOWN, Unable to get the status of the other supervisor.

I cannot ping the vmknic on control0

Also cannot take over by standby...

vsm-n1kv# system switchover

Failed to switch over (standby not ready to takeover)

[Dragomir]

I am seeing this in vsphere. The vlan on the control nics is showing vlan1. what is causing this?