Secondary VSM doesn't connect with Primary

naterevello · ‎01-05-2016

Environment: I'm currently using a dcloud lab for setting up a 1000v. I want to build the implementation 'from the ground up' or as close to it as I can.

dCloud Lab Name: Cisco Nexus 1000V: Installing Cisco VSM and VEM for VMWare Lab v1. The lab under the PEC is more of what I need, but it's closed for maintenance until 1/10/15. dCloud labs seem out of date since they prefer the installer app and version 4.2.

After running through that lab a few times I've decided to install the newest version of N1kv ( System 5.2(1)SV3(1.5a) ). I got the primary up and running. I configured the port profiles. I installed the VEM on the hosts, and migrated the hosts to the 1000v.

The trouble that I have run into is when I am setting up a secondary VSM for HA. I'm not sure if it's the lab environment that's the limitation, or if it's a configuration error on my part.

I did enter 'system redundancy role primary/secondary' on the respective VSM's. The secondary VSM didn't restart like it has in other Labs/environments. The other environment/dcloud lab that I tried has the VSM already installed, the lab is just for creating/managing the port profiles.

Primary VSM show commands:

N1kv# sh version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html

Copyright (c) 2002-2015, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public

License. A copy of the license is available at

http://www.gnu.org/licenses/gpl.html.

Software

kickstart: version 5.2(1)SV3(1.5a)

system:    version 5.2(1)SV3(1.5a)

kickstart image file is: bootflash:///n1000v-dk9-kickstart.5.2.1.SV3.1.5a.bin

kickstart compile time: 8/15/2015 6:00:00 [08/15/2015 13:39:52]

system image file is:    bootflash:///n1000v-dk9.5.2.1.SV3.1.5a.bin

system compile time:     8/15/2015 6:00:00 [08/15/2015 15:22:16]

Hardware

cisco Nexus 1000V Chassis ("Virtual Supervisor Module")

Intel(R) Xeon(R) CPU E7- 283 with 4126584 kB of memory.

Processor Board ID T5056813E21

Device name: N1kv

bootflash:    2332296 kB

System uptime is 0 days, 2 hours, 8 minutes, 30 seconds

Kernel uptime is 0 day(s), 2 hour(s), 9 minute(s), 7 second(s)

plugin

Core Plugin, Ethernet Plugin, Virtualization Plugin

Reset reason

1) Time: Sat Aug 15 08:27:53 2015

Reason: Reset Requested by CLI command reload

N1kv# sh svs domain

SVS domain config:

Domain id:    101

Control vlan: NA

Packet vlan:   NA

L2/L3 Control mode: L3

Switch guid: cf16c21c-aa93-428f-b98d-b6dce279281e

L3 control interface: mgmt0

Status: Config push to Management Server successful.

Control type multicast: No

L3Sec Status: Enabled

Note: Control VLAN and Packet VLAN are not used in L3 mode

N1kv# sh svs connections

connection vcenter:

    ip address: 198.18.133.211

    remote port: 80

    protocol: vmware-vim https

    certificate: default

    datacenter name: dCloud-DC

    admin:

    max-ports: 8192

    DVS uuid: 33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c

    config status: Enabled

    operational status: Connected

    sync status: Complete

    version: VMware vCenter Server 5.5.0 build-1312298

    vc-uuid: 67461318-8FFD-4EC1-8638-62D32F7285D7

    ssl-cert: self-signed or not authenticated

N1kv# sh module

Mod Ports Module-Type                       Model               Status

--- ----- -------------------------------- ------------------ ------------

1    0      Virtual Supervisor Module         Nexus1000V          active *

3    1022   Virtual Ethernet Module           NA                  ok

4    1022   Virtual Ethernet Module           NA                  ok

Mod Sw                  Hw

--- ------------------ ------------------------------------------------

1    5.2(1)SV3(1.5a)     0.0

3    5.2(1)SV3(1.5a)     VMware ESXi 5.5.0 Releasebuild-1331820 (3.2)

4    5.2(1)SV3(1.5a)     VMware ESXi 5.5.0 Releasebuild-1331820 (3.2)

Mod Server-IP        Server-UUID                           Server-Name

--- --------------- ------------------------------------ --------------------

1    198.18.133.40    NA                                    NA

3    198.18.133.31    422025f7-043a-87f5-c403-5b9efdf66764 vesx1.dcloud.cisco.com

4    198.18.133.32    4220955f-2062-8e3e-04b8-0000831108e7 vesx2.dcloud.cisco.com

* this terminal session

N1kv# sh int virtual

-------------------------------------------------------------------------------

Port        Adapter        Owner                    Mod Host

-------------------------------------------------------------------------------

Veth1       vmk0           VMware VMkernel          3   vesx1.dcloud.cisco.com

Veth2       vmk4           VMware VMkernel          3   vesx1.dcloud.cisco.com

Veth3       Net Adapter 1 N1kv-VSM-P               3   vesx1.dcloud.cisco.com

Veth4       Net Adapter 2 N1kv-VSM-P               3   vesx1.dcloud.cisco.com

Veth5       Net Adapter 3 N1kv-VSM-P               3   vesx1.dcloud.cisco.com

Veth6       vmk0           VMware VMkernel          4   vesx2.dcloud.cisco.com

Veth7       vmk4           VMware VMkernel          4   vesx2.dcloud.cisco.com

Veth8       Net Adapter 1 N1kv-VSM-S               4   vesx2.dcloud.cisco.com

Veth9       Net Adapter 2 N1kv-VSM-S               4   vesx2.dcloud.cisco.com

Veth10      Net Adapter 3 N1kv-VSM-S               4   vesx2.dcloud.cisco.com

N1kv# sh running-config

!Command: show running-config

!Time: Tue Jan 5 20:44:53 2016

version 5.2(1)SV3(1.5a)

hostname N1kv

no feature telnet

username admin password 5 $1$e/9mVYDR$lpRLU0EPoY9AApRAG/h3P. role network-admin

username admin keypair generate rsa

banner motd #Nexus 1000v Switch

#

ssh key rsa 2048

ip domain-lookup

ip host N1kv 198.18.133.40

errdisable recovery cause failed-port-state

vem 3

host id 422025f7-043a-87f5-c403-5b9efdf66764

vem 4

host id 4220955f-2062-8e3e-04b8-0000831108e7

snmp-server user admin network-admin auth md5 0xcab31b6c2edfee619396ff3266cc3970 priv 0xcab31b6c2edfee61

9396ff3266cc3970 localizedkey

rmon event 1 log trap public description FATAL(1) owner PMON@FATAL

rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL

rmon event 3 log trap public description ERROR(3) owner PMON@ERROR

rmon event 4 log trap public description WARNING(4) owner PMON@WARNING

rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO

vrf context management

ip route 0.0.0.0/0 198.18.128.1

vlan 1,10-12,111

vlan 10

name Management-vMotion

vlan 11

name Data-Network

vlan 12

name NFS

vlan 111

name PVLAN-Secondary

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

shutdown

description Port-group created for Nexus 1000V internal usage. Do not use.

state enabled

vmware port-group

port-profile type vethernet Unused_Or_Quarantine_Veth

shutdown

description Port-group created for Nexus 1000V internal usage. Do not use.

state enabled

vmware port-group

port-profile type ethernet n1kv_mgmt-uplink

switchport mode access

switchport access vlan 10

no shutdown

system vlan 10

state enabled

vmware port-group

port-profile type ethernet nfs-uplink

switchport mode access

switchport access vlan 12

no shutdown

system vlan 12

state enabled

vmware port-group

port-profile type ethernet data-uplink

switchport mode access

switchport access vlan 11

channel-group auto mode on mac-pinning

no shutdown

state enabled

vmware port-group

port-profile type vethernet n1kv_mgmt_vlan

switchport mode access

switchport access vlan 10

no shutdown

capability l3control

system vlan 10

state enabled

vmware port-group

port-profile type vethernet nfs_vlan

switchport mode access

switchport access vlan 12

no shutdown

system vlan 12

state enabled

vmware port-group

port-profile type vethernet vsm-control-packet

switchport mode access

switchport access vlan 1

no shutdown

state enabled

vmware port-group

port-profile type vethernet vsm-mgmt0

switchport mode access

switchport access vlan 10

no shutdown

system vlan 10

state enabled

vmware port-group

port-profile type vethernet VM-Client

switchport mode access

switchport access vlan 11

no shutdown

state enabled

vmware port-group

interface port-channel1

inherit port-profile data-uplink

vem 3

interface port-channel2

inherit port-profile data-uplink

vem 4

interface mgmt0

ip address 198.18.133.40/18

interface Vethernet1

inherit port-profile n1kv_mgmt_vlan

description VMware VMkernel, vmk0

vmware dvport 32 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.56A0.38BE

interface Vethernet2

inherit port-profile nfs_vlan

description VMware VMkernel, vmk4

vmware dvport 64 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.5660.64CA

interface Vethernet3

inherit port-profile vsm-control-packet

description N1kv-VSM-P, Network Adapter 1

vmware dvport 96 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.5681.636F

interface Vethernet4

inherit port-profile vsm-mgmt0

description N1kv-VSM-P, Network Adapter 2

vmware dvport 128 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.5681.3E21

interface Vethernet5

inherit port-profile vsm-control-packet

description N1kv-VSM-P, Network Adapter 3

vmware dvport 97 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.5681.2411

interface Vethernet6

inherit port-profile n1kv_mgmt_vlan

description VMware VMkernel, vmk0

vmware dvport 33 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.566C.C9B4

interface Vethernet7

inherit port-profile nfs_vlan

description VMware VMkernel, vmk4

vmware dvport 65 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.5660.CA92

interface Vethernet8

inherit port-profile vsm-control-packet

description N1kv-VSM-S, Network Adapter 1

vmware dvport 98 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.5681.326A

interface Vethernet9

inherit port-profile vsm-mgmt0

description N1kv-VSM-S, Network Adapter 2

vmware dvport 129 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.5681.DDB7

interface Vethernet10

inherit port-profile vsm-control-packet

description N1kv-VSM-S, Network Adapter 3

vmware dvport 99 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

vmware vm mac 0050.5681.B467

interface Ethernet3/1

inherit port-profile n1kv_mgmt-uplink

interface Ethernet3/2

inherit port-profile data-uplink

interface Ethernet3/3

inherit port-profile data-uplink

interface Ethernet3/4

inherit port-profile data-uplink

interface Ethernet3/5

inherit port-profile nfs-uplink

interface Ethernet4/1

inherit port-profile n1kv_mgmt-uplink

interface Ethernet4/2

inherit port-profile data-uplink

interface Ethernet4/3

inherit port-profile data-uplink

interface Ethernet4/4

inherit port-profile data-uplink

interface Ethernet4/5

inherit port-profile nfs-uplink

interface control0

line console

line vty

boot kickstart bootflash:/n1000v-dk9-kickstart.5.2.1.SV3.1.5a.bin sup-1

boot system bootflash:/n1000v-dk9.5.2.1.SV3.1.5a.bin sup-1

boot kickstart bootflash:/n1000v-dk9-kickstart.5.2.1.SV3.1.5a.bin sup-2

boot system bootflash:/n1000v-dk9.5.2.1.SV3.1.5a.bin sup-2

svs-domain

domain id 101

control vlan 1

packet vlan 1

svs mode L3 interface mgmt0

switch-guid cf16c21c-aa93-428f-b98d-b6dce279281e

enable l3sec

svs connection vcenter

protocol vmware-vim

remote ip address 198.18.133.211 port 80

vmware dvs uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c" datacenter-name dCloud-DC

max-ports 12000

connect

vservice global type vsg

no tcp state-checks invalid-ack

no tcp state-checks seq-past-window

no tcp state-checks window-variation

no bypass asa-traffic

no l3-frag

vservice global

idle-timeout

    tcp 30

    udp 4

    icmp 4

    layer-3 4

    layer-2 2

nsc-policy-agent

registration-ip 0.0.0.0

shared-secret **********

log-level

N1kv# sh port-profile usage

port-profile data-uplink

port-channel1

port-channel2

Ethernet3/2

Ethernet3/3

Ethernet3/4

Ethernet4/2

Ethernet4/3

Ethernet4/4

port-profile n1kv_mgmt-uplink

Ethernet3/1

Ethernet4/1

port-profile n1kv_mgmt_vlan

Vethernet1

Vethernet6

port-profile nfs-uplink

Ethernet3/5

Ethernet4/5

port-profile nfs_vlan

Vethernet2

Vethernet7

port-profile Unused_Or_Quarantine_Uplink

port-profile Unused_Or_Quarantine_Veth

port-profile VM-Client

port-profile vsm-control-packet

Vethernet3

Vethernet5

Vethernet8

Vethernet10

port-profile vsm-mgmt0

Vethernet4

Vethernet9

N1kv# sh system redundancy status

Redundancy role

---------------

      administrative:   primary

         operational:   primary

Redundancy mode

---------------

      administrative:   HA

         operational:   None

This supervisor (sup-1)

-----------------------

    Redundancy state:   Active

    Supervisor state:   Active

      Internal state:   Active with no standby

Other supervisor (sup-2)

------------------------

    Redundancy state:   Not present

Peer Sup Mac Adddreses Learnt

--------------------------------------------

   Control Interface:   00:50:56:81:06:14

      Mgmt Interface:   00:50:56:81:d1:ca

HA Packet Drops Due to Domain id Collision

--------------------------------------------

   Control Interface:   11001

      Mgmt Interface:   2581

-------------------------------------------------------------------------

IMPORTANT NOTE: Please compare Peer Sup MAC addresses learnt above

with the actual Peer Sup's MAC addresses. If they are not same, execute

"peer-sup mac-addresses clear" on this VSM to form HA again

-------------------------------------------------------------------------

For the secondary VSM, I have to go through the console, so I can't copy/paste. When I run the setup command and choose primary I get the following error (See attachment named: VSM secondary setup.PNG): SIOCSIFFLAGS: Permission denied I have not found a solution for this error.

The secondary VSM reloads and then I'm back to a VSM that acts like it's isolated. When I run sh module I get a public IP listed as the Server-IP. This is what makes me think there's some sort of limitation with the gear. This is also where I want another set of eyes, since maybe this solution is obvious to someone else with more experience. See VSM secondary module.PNG for a screenshot. I'm unable to ping the public IP for the secondary VSM.

I don't have access to the upstream switches. The VSM's were installed by me via OVA deploy.

For a screenshot of the VM nic settings, see Vcenter VM Ints.PNG Both the primary and secondary have the same settings.

Let me know if you need to see any other settings in Vcenter or need some more show commands.

Thanks!

naterevello · ‎01-05-2016

A follow-up questions is; How would I further troubleshoot, or verify, the connectivity?

James Clifford · ‎03-25-2016

Hi mate - in no particular order.....

Does it work if you put them in Layer 2 mode?
is port-profile type ethernet n1kv_mgmt-uplink on VSM 2 in the same vlan?
while in Layer 3 mode - can the all mgmt interfaces ping each other? including the vmware connection, gateway and VEMs?
Try this: enter the command "system redundany role primary" on the primary vsm and "system redundancy role secondary" on the secondary then run setup again.

Is this a lab environment?

Thanks

James