cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Cisco Community Designated VIP Class of 2020

137202
Views
1
Helpful
2
Replies
Highlighted
Not applicable

Secondary VSM doesn't connect with Primary

Environment: I'm currently using a dcloud lab for setting up a 1000v.  I want to build the implementation 'from the ground up' or as close to it as I can.

dCloud Lab Name: Cisco Nexus 1000V: Installing Cisco VSM and VEM for VMWare Lab v1.  The lab under the PEC is more of what I need, but it's closed for maintenance until 1/10/15.  dCloud labs seem out of date since they prefer the installer app and version 4.2.


After running through that lab a few times I've decided to install the newest version of N1kv ( System 5.2(1)SV3(1.5a) ).  I got the primary up and running.  I configured the port profiles.  I installed the VEM on the hosts, and migrated the hosts to the 1000v.

The trouble that I have run into is when I am setting up a secondary VSM for HA.  I'm not sure if it's the lab environment that's the limitation, or if it's a configuration error on my part.

I did enter 'system redundancy role primary/secondary' on the respective VSM's.  The secondary VSM didn't restart like it has in other Labs/environments.  The other environment/dcloud lab that I tried has the VSM already installed, the lab is just for creating/managing the port profiles.

Primary VSM show commands:

N1kv# sh version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html

Copyright (c) 2002-2015, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public

License. A copy of the license is available at

http://www.gnu.org/licenses/gpl.html.

Software

  kickstart: version 5.2(1)SV3(1.5a)

  system:    version 5.2(1)SV3(1.5a)

  kickstart image file is: bootflash:///n1000v-dk9-kickstart.5.2.1.SV3.1.5a.bin

  kickstart compile time:  8/15/2015 6:00:00 [08/15/2015 13:39:52]

  system image file is:    bootflash:///n1000v-dk9.5.2.1.SV3.1.5a.bin

  system compile time:     8/15/2015 6:00:00 [08/15/2015 15:22:16]

Hardware

  cisco Nexus 1000V Chassis ("Virtual Supervisor Module")

  Intel(R) Xeon(R) CPU E7- 283 with 4126584 kB of memory.

  Processor Board ID T5056813E21

  Device name: N1kv

  bootflash:    2332296 kB

System uptime is 0 days, 2 hours, 8 minutes, 30 seconds

Kernel uptime is 0 day(s), 2 hour(s), 9 minute(s), 7 second(s)

plugin

  Core Plugin, Ethernet Plugin, Virtualization Plugin

Reset reason

1) Time: Sat Aug 15 08:27:53 2015

Reason: Reset Requested by CLI command reload

N1kv# sh svs domain

SVS domain config:

  Domain id:    101

  Control vlan:  NA

  Packet vlan:   NA

  L2/L3 Control mode: L3

  Switch guid: cf16c21c-aa93-428f-b98d-b6dce279281e

  L3 control interface: mgmt0

  Status: Config push to Management Server successful.

  Control type multicast: No

  L3Sec Status: Enabled

Note: Control VLAN and Packet VLAN are not used in L3 mode

N1kv# sh svs connections

connection vcenter:

    ip address: 198.18.133.211

    remote port: 80

    protocol: vmware-vim https

    certificate: default

    datacenter name: dCloud-DC

    admin:

    max-ports: 8192

    DVS uuid: 33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c

    config status: Enabled

    operational status: Connected

    sync status: Complete

    version: VMware vCenter Server 5.5.0 build-1312298

    vc-uuid: 67461318-8FFD-4EC1-8638-62D32F7285D7

    ssl-cert: self-signed or not authenticated

N1kv# sh module

Mod  Ports  Module-Type                       Model               Status

---  -----  --------------------------------  ------------------  ------------

1    0      Virtual Supervisor Module         Nexus1000V          active *

3    1022   Virtual Ethernet Module           NA                  ok

4    1022   Virtual Ethernet Module           NA                  ok

Mod  Sw                  Hw

---  ------------------  ------------------------------------------------

1    5.2(1)SV3(1.5a)     0.0

3    5.2(1)SV3(1.5a)     VMware ESXi 5.5.0 Releasebuild-1331820 (3.2)

4    5.2(1)SV3(1.5a)     VMware ESXi 5.5.0 Releasebuild-1331820 (3.2)

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

1    198.18.133.40    NA                                    NA

3    198.18.133.31    422025f7-043a-87f5-c403-5b9efdf66764  vesx1.dcloud.cisco.com

4    198.18.133.32    4220955f-2062-8e3e-04b8-0000831108e7  vesx2.dcloud.cisco.com

* this terminal session

N1kv# sh int virtual

-------------------------------------------------------------------------------

Port        Adapter        Owner                    Mod Host

-------------------------------------------------------------------------------

Veth1       vmk0           VMware VMkernel          3   vesx1.dcloud.cisco.com

Veth2       vmk4           VMware VMkernel          3   vesx1.dcloud.cisco.com

Veth3       Net Adapter 1  N1kv-VSM-P               3   vesx1.dcloud.cisco.com

Veth4       Net Adapter 2  N1kv-VSM-P               3   vesx1.dcloud.cisco.com

Veth5       Net Adapter 3  N1kv-VSM-P               3   vesx1.dcloud.cisco.com

Veth6       vmk0           VMware VMkernel          4   vesx2.dcloud.cisco.com

Veth7       vmk4           VMware VMkernel          4   vesx2.dcloud.cisco.com

Veth8       Net Adapter 1  N1kv-VSM-S               4   vesx2.dcloud.cisco.com

Veth9       Net Adapter 2  N1kv-VSM-S               4   vesx2.dcloud.cisco.com

Veth10      Net Adapter 3  N1kv-VSM-S               4   vesx2.dcloud.cisco.com

N1kv# sh running-config

!Command: show running-config

!Time: Tue Jan  5 20:44:53 2016

version 5.2(1)SV3(1.5a)

hostname N1kv

no feature telnet

username admin password 5 $1$e/9mVYDR$lpRLU0EPoY9AApRAG/h3P.  role network-admin

username admin keypair generate rsa

banner motd #Nexus 1000v Switch

#

ssh key rsa 2048

ip domain-lookup

ip host N1kv 198.18.133.40

errdisable recovery cause failed-port-state

vem 3

  host id 422025f7-043a-87f5-c403-5b9efdf66764

vem 4

  host id 4220955f-2062-8e3e-04b8-0000831108e7

snmp-server user admin network-admin auth md5 0xcab31b6c2edfee619396ff3266cc3970 priv 0xcab31b6c2edfee61

9396ff3266cc3970 localizedkey

rmon event 1 log trap public description FATAL(1) owner PMON@FATAL

rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL

rmon event 3 log trap public description ERROR(3) owner PMON@ERROR

rmon event 4 log trap public description WARNING(4) owner PMON@WARNING

rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO

vrf context management

  ip route 0.0.0.0/0 198.18.128.1

vlan 1,10-12,111

vlan 10

  name Management-vMotion

vlan 11

  name Data-Network

vlan 12

  name NFS

vlan 111

  name PVLAN-Secondary

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

  shutdown

  description Port-group created for Nexus 1000V internal usage. Do not use.

  state enabled

  vmware port-group

port-profile type vethernet Unused_Or_Quarantine_Veth

  shutdown

  description Port-group created for Nexus 1000V internal usage. Do not use.

  state enabled

  vmware port-group

port-profile type ethernet n1kv_mgmt-uplink

  switchport mode access

  switchport access vlan 10

  no shutdown

  system vlan 10

  state enabled

  vmware port-group

port-profile type ethernet nfs-uplink

  switchport mode access

  switchport access vlan 12

  no shutdown

  system vlan 12

  state enabled

  vmware port-group

port-profile type ethernet data-uplink

  switchport mode access

  switchport access vlan 11

  channel-group auto mode on mac-pinning

  no shutdown

  state enabled

  vmware port-group

port-profile type vethernet n1kv_mgmt_vlan

  switchport mode access

  switchport access vlan 10

  no shutdown

  capability l3control

  system vlan 10

  state enabled

  vmware port-group

port-profile type vethernet nfs_vlan

  switchport mode access

  switchport access vlan 12

  no shutdown

  system vlan 12

  state enabled

  vmware port-group

port-profile type vethernet vsm-control-packet

  switchport mode access

  switchport access vlan 1

  no shutdown

  state enabled

  vmware port-group

port-profile type vethernet vsm-mgmt0

  switchport mode access

  switchport access vlan 10

  no shutdown

  system vlan 10

  state enabled

  vmware port-group

port-profile type vethernet VM-Client

  switchport mode access

  switchport access vlan 11

  no shutdown

  state enabled

  vmware port-group

interface port-channel1

  inherit port-profile data-uplink

  vem 3

interface port-channel2

  inherit port-profile data-uplink

  vem 4

interface mgmt0

  ip address 198.18.133.40/18

interface Vethernet1

  inherit port-profile n1kv_mgmt_vlan

  description VMware VMkernel, vmk0

  vmware dvport 32 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.56A0.38BE

interface Vethernet2

  inherit port-profile nfs_vlan

  description VMware VMkernel, vmk4

  vmware dvport 64 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.5660.64CA

interface Vethernet3

  inherit port-profile vsm-control-packet

  description N1kv-VSM-P, Network Adapter 1

  vmware dvport 96 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.5681.636F

interface Vethernet4

  inherit port-profile vsm-mgmt0

  description N1kv-VSM-P, Network Adapter 2

  vmware dvport 128 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.5681.3E21

interface Vethernet5

  inherit port-profile vsm-control-packet

  description N1kv-VSM-P, Network Adapter 3

  vmware dvport 97 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.5681.2411

interface Vethernet6

  inherit port-profile n1kv_mgmt_vlan

  description VMware VMkernel, vmk0

  vmware dvport 33 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.566C.C9B4

interface Vethernet7

  inherit port-profile nfs_vlan

  description VMware VMkernel, vmk4

  vmware dvport 65 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.5660.CA92

interface Vethernet8

  inherit port-profile vsm-control-packet

  description N1kv-VSM-S, Network Adapter 1

  vmware dvport 98 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.5681.326A

interface Vethernet9

  inherit port-profile vsm-mgmt0

  description N1kv-VSM-S, Network Adapter 2

  vmware dvport 129 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.5681.DDB7

interface Vethernet10

  inherit port-profile vsm-control-packet

  description N1kv-VSM-S, Network Adapter 3

  vmware dvport 99 dvswitch uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c"

  vmware vm mac 0050.5681.B467

interface Ethernet3/1

  inherit port-profile n1kv_mgmt-uplink

interface Ethernet3/2

  inherit port-profile data-uplink

interface Ethernet3/3

  inherit port-profile data-uplink

interface Ethernet3/4

  inherit port-profile data-uplink

interface Ethernet3/5

  inherit port-profile nfs-uplink

interface Ethernet4/1

  inherit port-profile n1kv_mgmt-uplink

interface Ethernet4/2

  inherit port-profile data-uplink

interface Ethernet4/3

  inherit port-profile data-uplink

interface Ethernet4/4

  inherit port-profile data-uplink

interface Ethernet4/5

  inherit port-profile nfs-uplink

interface control0

line console

line vty

boot kickstart bootflash:/n1000v-dk9-kickstart.5.2.1.SV3.1.5a.bin sup-1

boot system bootflash:/n1000v-dk9.5.2.1.SV3.1.5a.bin sup-1

boot kickstart bootflash:/n1000v-dk9-kickstart.5.2.1.SV3.1.5a.bin sup-2

boot system bootflash:/n1000v-dk9.5.2.1.SV3.1.5a.bin sup-2

svs-domain

  domain id 101

  control vlan 1

  packet vlan 1

  svs mode L3 interface mgmt0

  switch-guid cf16c21c-aa93-428f-b98d-b6dce279281e

  enable l3sec

svs connection vcenter

  protocol vmware-vim

  remote ip address 198.18.133.211 port 80

  vmware dvs uuid "33 4b 01 50 c4 35 a8 98-47 06 70 50 d9 c6 3f 9c" datacenter-name dCloud-DC

  max-ports 12000

  connect

vservice global type vsg

  no tcp state-checks invalid-ack

  no tcp state-checks seq-past-window

  no tcp state-checks window-variation

  no bypass asa-traffic

  no l3-frag

vservice global

  idle-timeout

    tcp 30

    udp 4

    icmp 4

    layer-3 4

    layer-2 2

nsc-policy-agent

  registration-ip 0.0.0.0

  shared-secret **********

  log-level

N1kv# sh port-profile usage

port-profile data-uplink

port-channel1

port-channel2

Ethernet3/2

Ethernet3/3

Ethernet3/4

Ethernet4/2

Ethernet4/3

Ethernet4/4

port-profile n1kv_mgmt-uplink

Ethernet3/1

Ethernet4/1

port-profile n1kv_mgmt_vlan

Vethernet1

Vethernet6

port-profile nfs-uplink

Ethernet3/5

Ethernet4/5

port-profile nfs_vlan

Vethernet2

Vethernet7

port-profile Unused_Or_Quarantine_Uplink

port-profile Unused_Or_Quarantine_Veth

port-profile VM-Client

port-profile vsm-control-packet

Vethernet3

Vethernet5

Vethernet8

Vethernet10

port-profile vsm-mgmt0

Vethernet4

Vethernet9

N1kv# sh system redundancy status

Redundancy role

---------------

      administrative:   primary

         operational:   primary

Redundancy mode

---------------

      administrative:   HA

         operational:   None

This supervisor (sup-1)

-----------------------

    Redundancy state:   Active

    Supervisor state:   Active

      Internal state:   Active with no standby

Other supervisor (sup-2)

------------------------

    Redundancy state:   Not present

Peer Sup Mac Adddreses Learnt

--------------------------------------------

   Control Interface:   00:50:56:81:06:14

      Mgmt Interface:   00:50:56:81:d1:ca

HA Packet Drops Due to Domain id Collision

--------------------------------------------

   Control Interface:   11001

      Mgmt Interface:   2581

-------------------------------------------------------------------------

IMPORTANT NOTE: Please compare Peer Sup MAC addresses learnt above

with the actual Peer Sup's MAC addresses. If they are not same, execute

"peer-sup mac-addresses clear" on this VSM to form HA again

-------------------------------------------------------------------------

For the secondary VSM, I have to go through the console, so I can't copy/paste.  When I run the setup command and choose primary I get the following error (See attachment named: VSM secondary setup.PNG): SIOCSIFFLAGS: Permission denied  I have not found a solution for this error.

The secondary VSM reloads and then I'm back to a VSM that acts like it's isolated.  When I run sh module I get a public IP listed as the Server-IP.  This is what makes me think there's some sort of limitation with the gear.  This is also where I want another set of eyes, since maybe this solution is obvious to someone else with more experience.  See VSM secondary module.PNG for a screenshot.  I'm unable to ping the public IP for the secondary VSM.

I don't have access to the upstream switches.  The VSM's were installed by me via OVA deploy.

For a screenshot of the VM nic settings, see Vcenter VM Ints.PNG  Both the primary and secondary have the same settings.

Let me know if you need to see any other settings in Vcenter or need some more show commands.

Thanks!

Everyone's tags (5)
2 REPLIES 2
Not applicable

Re: Secondary VSM doesn't connect with Primary

A follow-up questions is; How would I further troubleshoot, or verify, the connectivity?

Beginner

Re: Secondary VSM doesn't connect with Primary

Hi mate - in no particular order.....

  1. Does it work if you put them in Layer 2 mode?
  2. is port-profile type ethernet n1kv_mgmt-uplink  on VSM 2 in the same vlan?
  3. while in Layer 3 mode - can the all mgmt interfaces ping each other? including the vmware connection, gateway and VEMs?
  4. Try this: enter the command "system redundany role primary" on the primary vsm and "system redundancy role secondary" on the secondary then run setup again.

Is this a lab environment?

Thanks

James

CreatePlease to create content
Content for Community-Ad
FusionCharts will render here