Solved: Layer3 mode Primary and Secondary VSM Communication problem

cisco4itec · ‎04-07-2013

I have a HA Pair setup and everything works fine when both my primary and secondary VSMs are on the same host. However, when I migrate either of my VSMs to another host Communication is lost.

Both hosts are on the same subnet and is able to ping each other. Subnet is 10.20.0.0/16

Any help would be much appreciated..

I also receive this message on my primary VSM:

%VEM_MGR-SLOT5-1-VEM_SYSLOG_ALERT: sfswitchdata : L3 Control and System VLAN configurations not applied on vethernet port. VMware Port[83886108] DVPortGroup[dvportgroup-36347]. L3 Control can be applied only on VMKernel port.

Show module on my VSM produces this output:

Mod Sw

Hw

--- ------------------ ------------------------------------------------

1	4.2(1)SV2(1.1a)	0.0
3	4.2(1)SV2(1.1a)	VMware ESXi 5.1.0 Releasebuild-1021289 (3.1)
4	4.2(1)SV2(1.1a)	VMware ESXi 5.1.0 Releasebuild-1021289 (3.1)
5	4.2(1)SV2(1.1a)	VMware ESXi 5.1.0 Releasebuild-1021289 (3.1)

Mod MAC-Address(es)

Serial-Num

--- -------------------------------------- ----------

1	00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA
3	02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA
4	02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA
5	02-00-0c-00-05-00 to 02-00-0c-00-05-80 NA

Mod Server-IP

Server-UUID

Server-Name

--- --------------- ------------------------------------ --------------------

1	10.20.255.1	NA	NA
3	10.20.2.10	33343934-3932-5355-4530-34354e344a4c xxxxxxx.xxxxxxxxxx.com
4	10.20.0.27	33343934-3932-5355-4530-34354e344a53 xxxxxxxx.xxxxxxxx.com
5	10.20.0.28	33343934-3932-5355-4530-34354e344b31 xxxxxxxxx.xxxxxxxx.com

my uplink profile:

port-profile system-uplinks

type: Ethernet

description: uplinks to management nics p1,2 (v0,1)

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode trunk

switchport trunk allowed vlan 20

no shutdown

evaluated config attributes:

switchport mode trunk

switchport trunk allowed vlan 20

no shutdown

assigned interfaces:

Ethernet3/1

Ethernet3/2

Ethernet4/1

Ethernet4/2

Ethernet5/1

Ethernet5/2

port-group: system-uplinks

system vlans: 20

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

My vethernet profiles:

port-profile mgmt

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 20

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 20

no shutdown

assigned interfaces:

Vethernet1

Vethernet3

Vethernet4

Vethernet6

Vethernet7

Vethernet13

Vethernet18

port-group: mgmt

system vlans: 20

capability l3control: yes

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile ctrl-pkt

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 20

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 20

no shutdown

assigned interfaces:

port-group: ctrl-pkt

system vlans: 20

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

lwatta · ‎04-09-2013

You only have 6 veth ports. If you have moved all your VSM connections and your vmk0 ports to the n1kv you should have 8 veth ports.

What I see is the following

vmk0 for host 1
vmk0 for host 2
vsm01 adapter 2
vsm02 adapter 1
vsm02 adapter 2
vsm02 adapter 3

You are missing connections for vsm01. You need to move vsm01 adapter 1 and 3 to the n1kv_mgmt port-profile. Heartbeats between the two VSMs occur over adapter 1. Since VSM01 adapter 1 does not seem to be attached to anything thats why HA is not fully working.

View solution in original post

cisco4itec · ‎04-07-2013

here is my show module output

Mod Ports Module-Type                       Model               Status
--- ----- -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
2    0      Virtual Supervisor Module                             powered-up
3    248    Virtual Ethernet Module           NA                  ok
4    248    Virtual Ethernet Module           NA                  ok
5    248    Virtual Ethernet Module           NA                  ok

lwatta · ‎04-08-2013

Errol,

Port-profiles with capbility l3control can only be assigned to vmk interfaces. So you can't assign a VM network interface to any port-profile with that setting. That's the reason you are getting the following error.

%VEM_MGR-SLOT5-1-VEM_SYSLOG_ALERT: sfswitchdata : L3 Control and System VLAN configurations not applied on vethernet port. VMware Port[83886108] DVPortGroup[dvportgroup-36347]. L3 Control can be applied only on VMKernel port.

Your standby VSM is not correctly joining, most likely the problem is that you are assigning your VSM mgmt interfaces to the "mgmt" port-profile which has l3control set. This is wrong because of the issue above.

I'm going to assume you are using L3 control and assigning your ESXi hosts to the mgmt port-profile. This is fine, but for the management interface of the VSMs you need to create a new port-profile that looks identical to your mgmt port-profile but take out the "capability l3control"

That should fix the problem

cisco4itec · ‎04-08-2013

Louis,

Thank you. I'll try your suggestion.

I do have one question, is it possible to have both layer3 control and Vmware management on the same vmk interface?

cisco4itec · ‎04-08-2013

Hi Louis,

I did as you suggested with no luck. I really help you or someone can help as I don't have a smartnet contract for my nexus 1000Vs

Thanks in advance.

I've uploaded a snapshot of vmware vds view as well.

here's what my new port-profile configs looks like:

port-profile n1kv_ctrl

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 20

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 20

no shutdown

assigned interfaces:

port-group: n1kv_ctrl

system vlans: 20

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile n1kv_mgmt

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 20

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 20

no shutdown

assigned interfaces:

Vethernet2

Vethernet4

port-group: n1kv_mgmt

system vlans: 20

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile n1kv_pkt

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 20

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 20

no shutdown

assigned interfaces:

port-group: n1kv_pkt

system vlans: 20

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile system-uplinks

type: Ethernet

description: uplinks to management nics p1,2 (v0,1)

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode trunk

switchport trunk allowed vlan 20

channel-group auto mode on

no shutdown

evaluated config attributes:

switchport mode trunk

switchport trunk allowed vlan 20

channel-group auto mode on

no shutdown

assigned interfaces:

port-channel1

port-channel2

Ethernet3/1

Ethernet3/2

Ethernet4/1

Ethernet4/2

port-group: system-uplinks

system vlans: 20

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

lwatta · ‎04-09-2013

How come your control and packet interfaces are not on the DVS? are they still connected on vSwitch?

You have channel-group auto mode on in the eth port-profile. This means you are doing port-channels on the upstream switches. Do you have port-channels defined on the uplink switches?

Lets make it really simple.

It's perfectly fine to put the control, packet, and mgmt interfaces of the VSMs in the same port-profile.

So put all your VSM interfaces in the n1kv_mgmt port-profile.

louis

cisco4itec · ‎04-09-2013

Ok I put all my VSM interfaces in the n1k_mgmt port-profile I also removed channel-group. Still nothing. However, I don't see this message anymore:

%VEM_MGR-SLOT5-1-VEM_SYSLOG_ALERT: sfswitchdata : L3 Control and System VLAN configurations not applied on vethernet port. VMware Port[83886108] DVPortGroup[dvportgroup-36347]. L3 Control can be applied only on VMKernel port.

But it seems my secondary vsm still can't find my primary VSM it's stuck at the loader loading:

Mod Ports Module-Type                       Model               Status
--- ----- -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
2    0      Virtual Supervisor Module                             powered-up
3    248    Virtual Ethernet Module           NA                  ok
4    248    Virtual Ethernet Module           NA                  ok

Mod Sw                  Hw
--- ------------------ ------------------------------------------------
1    4.2(1)SV2(1.1a)     0.0
3    4.2(1)SV2(1.1a)     VMware ESXi 5.1.0 Releasebuild-1021289 (3.1)
4    4.2(1)SV2(1.1a)     VMware ESXi 5.1.0 Releasebuild-1021289 (3.1)

Mod MAC-Address(es)                         Serial-Num
--- -------------------------------------- ----------
1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA
3    02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA
4    02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA

Mod Server-IP        Server-UUID                           Server-Name
--- --------------- ------------------------------------ --------------------
1    10.20.255.1      NA                                    NA
3    10.20.2.10       33343934-3932-5355-4530-34354e344a4c esx-r02u10
4    10.20.2.12       33343934-3932-5355-4530-34354e344b31 esx-r02u12

lwatta · ‎04-09-2013

Please attach a sho int brief and your running config.

louis

cisco4itec · ‎04-09-2013

n1kv-internal(config-port-prof)# sh int brief

--------------------------------------------------------------------------------

Port VRF Status IP Address Speed MTU

--------------------------------------------------------------------------------

mgmt0 -- up 10.20.255.1 1000 1500

--------------------------------------------------------------------------------

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

--------------------------------------------------------------------------------

Eth3/1 1 eth trunk up none 1000

Eth3/2 1 eth trunk up none 1000

Eth4/1 1 eth trunk up none 1000

Eth4/2 1 eth trunk up none 1000

--------------------------------------------------------------------------------

Port-channel VLAN Type Mode Status Reason Speed Protocol

Interface

--------------------------------------------------------------------------------

Po1 1 eth trunk down No operational members auto(I) none

Po2 1 eth trunk down No operational members auto(I) none

--------------------------------------------------------------------------------

Vethernet VLAN Type Mode Status Reason Speed

--------------------------------------------------------------------------------

Veth1 20 virt access up none auto

Veth2 20 virt access up none auto

Veth3 20 virt access up none auto

Veth4 20 virt access up none auto

Veth5 20 virt access up none auto

Veth6 20 virt access up none auto

--------------------------------------------------------------------------------

Port VRF Status IP Address Speed MTU

--------------------------------------------------------------------------------

control0 -- up -- 1000 1500

n1kv-internal(config-port-prof)# sh run

!Command: show running-config

!Time: Tue Apr 9 05:08:47 2013

version 4.2(1)SV2(1.1a)

svs switch edition essential

no feature telnet

username admin password 5 $1$j5bY17tU$FYUE3cR/.rzQx0CSdLq4C0 role network-admin

banner motd #Nexus 1000v Switch#

ip domain-lookup

ip host n1kv-internal 10.20.255.1

switchname n1kv-internal

errdisable recovery cause failed-port-state

vem 3

host vmware id 33343934-3932-5355-4530-34354e344a4c

vem 4

host vmware id 33343934-3932-5355-4530-34354e344b31

snmp-server user admin network-admin auth md5 0x70a68e683dd37faee4c18f978558f563 priv 0x70a68e683dd37faee4c18f978558f563 localizedkey

vrf context management

ip route 0.0.0.0/0 10.20.0.1

vlan 1,20

vlan 20

name mgmt-vlan

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type ethernet system-uplinks

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 20

no shutdown

system vlan 20

description uplinks to management nics p1,2 (v0,1)

state enabled

port-profile type vethernet L3vmkernel

capability l3control

vmware port-group

switchport mode access

switchport access vlan 20

no shutdown

system vlan 20

state enabled

port-profile type vethernet n1kv_ctrl

vmware port-group

switchport mode access

switchport access vlan 20

no shutdown

system vlan 20

state enabled

port-profile type vethernet n1kv_mgmt

vmware port-group

switchport mode access

switchport access vlan 20

no shutdown

system vlan 20

state enabled

port-profile type vethernet n1kv_pkt

vmware port-group

switchport mode access

switchport access vlan 20

no shutdown

system vlan 20

state enabled

vdc n1kv-internal id 1

limit-resource vlan minimum 16 maximum 2049

limit-resource monitor-session minimum 0 maximum 2

limit-resource vrf minimum 16 maximum 8192

limit-resource port-channel minimum 0 maximum 768

limit-resource u4route-mem minimum 1 maximum 1

limit-resource u6route-mem minimum 1 maximum 1

interface port-channel1

inherit port-profile system-uplinks

vem 3

interface port-channel2

inherit port-profile system-uplinks

vem 4

interface mgmt0

ip address 10.20.255.1/16

interface Vethernet1

inherit port-profile L3vmkernel

description VMware VMkernel, vmk0

vmware dvport 0 dvswitch uuid "6a 1e 27 50 fd fb 1d d4-a1 00 16 4e ea d5 37 3b"

vmware vm mac 1CC1.DE24.F48A

interface Vethernet2

inherit port-profile n1kv_mgmt

description n1kv-vsm01, Network Adapter 2

vmware dvport 100 dvswitch uuid "6a 1e 27 50 fd fb 1d d4-a1 00 16 4e ea d5 37 3b"

vmware vm mac 0050.56A7.02E9

interface Vethernet3

inherit port-profile L3vmkernel

description VMware VMkernel, vmk0

vmware dvport 10 dvswitch uuid "6a 1e 27 50 fd fb 1d d4-a1 00 16 4e ea d5 37 3b"

vmware vm mac 1CC1.DE24.E530

interface Vethernet4

inherit port-profile n1kv_mgmt

description n1kv-vsm02, Network Adapter 2

vmware dvport 102 dvswitch uuid "6a 1e 27 50 fd fb 1d d4-a1 00 16 4e ea d5 37 3b"

vmware vm mac 0050.56A7.4438

interface Vethernet5

inherit port-profile n1kv_mgmt

description n1kv-vsm02, Network Adapter 1

vmware dvport 101 dvswitch uuid "6a 1e 27 50 fd fb 1d d4-a1 00 16 4e ea d5 37 3b"

vmware vm mac 0050.56A7.7910

interface Vethernet6

inherit port-profile n1kv_mgmt

description n1kv-vsm02, Network Adapter 3

vmware dvport 103 dvswitch uuid "6a 1e 27 50 fd fb 1d d4-a1 00 16 4e ea d5 37 3b"

vmware vm mac 0050.56A7.7540

interface Ethernet3/1

inherit port-profile system-uplinks

interface Ethernet3/2

inherit port-profile system-uplinks

interface Ethernet4/1

inherit port-profile system-uplinks

interface Ethernet4/2

inherit port-profile system-uplinks

interface control0

clock timezone HST -10 0

line console

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-1

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-2

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-2

svs-domain

domain id 1

control vlan 1

packet vlan 1

svs mode L3 interface mgmt0

svs connection vcenter

protocol vmware-vim

remote ip address 10.10.0.200 port 80

vmware dvs uuid "6a 1e 27 50 fd fb 1d d4-a1 00 16 4e ea d5 37 3b" datacenter-name MY-DATACENTER

admin user n1kUser

max-ports 8192

connect

vservice global type vsg

tcp state-checks invalid-ack

tcp state-checks seq-past-window

no tcp state-checks window-variation

no bypass asa-traffic

vnm-policy-agent

registration-ip 0.0.0.0

shared-secret **********

log-level

lwatta · ‎04-09-2013

You only have 6 veth ports. If you have moved all your VSM connections and your vmk0 ports to the n1kv you should have 8 veth ports.

What I see is the following

vmk0 for host 1
vmk0 for host 2
vsm01 adapter 2
vsm02 adapter 1
vsm02 adapter 2
vsm02 adapter 3

You are missing connections for vsm01. You need to move vsm01 adapter 1 and 3 to the n1kv_mgmt port-profile. Heartbeats between the two VSMs occur over adapter 1. Since VSM01 adapter 1 does not seem to be attached to anything thats why HA is not fully working.

cisco4itec · ‎04-09-2013

Success!! Thanks Louis Problem was that I had my control and packet interface on my VSM01 pointing to another the Vswitch interfaces which are called n1kv_mgmt (why i thought it was using the correct port group.)

Thanks for your help with this..

I have one last question. I'd like to clean up my config. Is it mandatory that I keep my n1kv_ctrl and n1kv_pkt port-profiles since i'm using my n1kv_mgmt for all 3?

Thanks again for your help!!..

lwatta · ‎04-09-2013

Awesome. Glad you got it working.

You can delete the unused port-profiles.

louis