cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4949
Views
0
Helpful
5
Replies

VXLAN on UCS: IGMP with Catalyst 3750, Nexus 5548, Nexus 1000V

Hello Team,

My lab consists of Catalyst 3750 with SVI serving as the router, Nexus 5548 in vpc configuration, UCS in End-Host Mode and Nexus 1000V with feature segmentation (VXLAN) enabled.

I have two different VLANs for VXLAN (140, 141) to demonstrate connectivity across L3.

The hosts with VMKernel on VLAN 140 join the multicast group just fine.

The hosts with VMKernel on VLAN 141 do not join the multicast group.  So, VMs on these hosts cannot ping VMs on the VLAN 140 hosts, and they cannot even ping each other.

I turned on debug ip igmp on the L3 Switch, and the output shows a timeout when waiting for a report from VLAN 141:

Oct 15 08:57:34.201: IGMP(0): Send v2 general Query on Vlan140

Oct 15 08:57:34.201: IGMP(0): Set report delay time to 3.6 seconds for 224.0.1.40 on Vlan140

Oct 15 08:57:36.886: IGMP(0): Received v2 Report on Vlan140 from 172.16.66.2 for 239.1.1.1

Oct 15 08:57:36.886: IGMP(0): Received Group record for group 239.1.1.1, mode 2 from 172.16.66.2 for 0 sources

Oct 15 08:57:36.886: IGMP(0): Updating EXCLUDE group timer for 239.1.1.1

Oct 15 08:57:36.886: IGMP(0): MRT Add/Update Vlan140 for (*,239.1.1.1) by 0

Oct 15 08:57:38.270: IGMP(0): Send v2 Report for 224.0.1.40 on Vlan140

Oct 15 08:57:38.270: IGMP(0): Received v2 Report on Vlan140 from 172.16.66.1 for 224.0.1.40

Oct 15 08:57:38.270: IGMP(0): Received Group record for group 224.0.1.40, mode 2 from 172.16.66.1 for 0 sources

Oct 15 08:57:38.270: IGMP(0): Updating EXCLUDE group timer for 224.0.1.40

Oct 15 08:57:38.270: IGMP(0): MRT Add/Update Vlan140 for (*,224.0.1.40) by 0

Oct 15 08:57:51.464: IGMP(0): Send v2 general Query on Vlan141 <----- It just hangs here until timeout and goes back to vlan140

Oct 15 08:58:35.107: IGMP(0): Send v2 general Query on Vlan140

Oct 15 08:58:35.107: IGMP(0): Set report delay time to 0.3 seconds for 224.0.1.40 on Vlan140

Oct 15 08:58:35.686: IGMP(0): Received v2 Report on Vlan140 from 172.16.66.2 for 239.1.1.1

Oct 15 08:58:35.686: IGMP(0): Received Group record for group 239.1.1.1, mode 2 from 172.16.66.2 for 0 sources

Oct 15 08:58:35.686: IGMP(0): Updating EXCLUDE group timer for 239.1.1.1

Oct 15 08:58:35.686: IGMP(0): MRT Add/Update Vlan140 for (*,239.1.1.1) by 0

if I do a show ip igmp interface, I get the report that there are no joins for vlan 141:

Vlan140 is up, line protocol is up

  Internet address is 172.16.66.1/26

  IGMP is enabled on interface

  Current IGMP host version is 2

  Current IGMP router version is 2

  IGMP query interval is 60 seconds

  IGMP configured query interval is 60 seconds

  IGMP querier timeout is 120 seconds

  IGMP configured querier timeout is 120 seconds

  IGMP max query response time is 10 seconds

  Last member query count is 2

  Last member query response interval is 1000 ms

  Inbound IGMP access group is not set

  IGMP activity: 2 joins, 0 leaves

  Multicast routing is enabled on interface

  Multicast TTL threshold is 0

  Multicast designated router (DR) is 172.16.66.1 (this system)

  IGMP querying router is 172.16.66.1 (this system)

  Multicast groups joined by this system (number of users):

      224.0.1.40(1)

Vlan141 is up, line protocol is up

  Internet address is 172.16.66.65/26

  IGMP is enabled on interface

  Current IGMP host version is 2

  Current IGMP router version is 2

  IGMP query interval is 60 seconds

  IGMP configured query interval is 60 seconds

  IGMP querier timeout is 120 seconds

  IGMP configured querier timeout is 120 seconds

  IGMP max query response time is 10 seconds

  Last member query count is 2

  Last member query response interval is 1000 ms

  Inbound IGMP access group is not set

  IGMP activity: 0 joins, 0 leaves

  Multicast routing is enabled on interface

  Multicast TTL threshold is 0

  Multicast designated router (DR) is 172.16.66.65 (this system)

  IGMP querying router is 172.16.66.65 (this system)

  No multicast groups joined by this system

Is there a way to verify why the hosts on VLAN 141 are not successfully joining?  port-profile configuration on the 1000V for the vlan 140 and vlan 141 uplinks and vmkernel are identical except for the different vlan numbers.

Thanks,

Trevor

====================================================================================== If my answers have been helpful in any way, please rate accordingly. Check out my blog for useful vSphere & UCS tips: www.vmtrooper.com
2 Accepted Solutions

Accepted Solutions

abbharga
Level 4
Level 4

Hi Trevor,

Once quick thing to check would be the igmp config for both the vlans.

where do you have the querier configure for the vlan 140 and 141?

are there changes of VXLAN transport traversing routers? if so you would need multicast routing enabled as well.

Thanks!

./abhinav

View solution in original post

Hello Trevor,

Could you please attach 'sh run's from N1k & 3750?

One a VLAN 141 host collect:

vemcmd show port

vemcmd show port old

vemcmd show vxlan interfaces

Matthew

View solution in original post

5 Replies 5

abbharga
Level 4
Level 4

Hi Trevor,

Once quick thing to check would be the igmp config for both the vlans.

where do you have the querier configure for the vlan 140 and 141?

are there changes of VXLAN transport traversing routers? if so you would need multicast routing enabled as well.

Thanks!

./abhinav

Hello Abhinav,

Querier for both VLANs is configured on the 3750.  I only have one in my setup.  So, i don't have to worry about stackwise configs.

VXLAN traffic is not traversing routers.  I have a single L3 device in my environment.  The VLAN traffic is routed to the 3750.

What is confusing me that the hosts on VLAN 140 respond to queries without any problem.  It's VLAN 141 that hangs until a timeout.

Anything else I can provide for troubleshoot?

====================================================================================== If my answers have been helpful in any way, please rate accordingly. Check out my blog for useful vSphere & UCS tips: www.vmtrooper.com

Hello Trevor,

Could you please attach 'sh run's from N1k & 3750?

One a VLAN 141 host collect:

vemcmd show port

vemcmd show port old

vemcmd show vxlan interfaces

Matthew

Hello Matthew,

I didn't see a way to attach a file.  so, I'm pasting the requested output:

3750

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2012.10.15 11:45:05 =~=~=~=~=~=~=~=~=~=~=~=

sh run

Building configuration...

Current configuration : 8989 bytes

!

! Last configuration change at 04:22:24 EDT Mon Oct 15 2012 by admin

! NVRAM config last updated at 05:51:05 EDT Mon Oct 15 2012 by admin

!

version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname vb3750stack

!

boot-start-marker

boot-end-marker

!

no aaa new-model

clock timezone EST -5

clock summer-time EDT recurring

system mtu routing 1600

ip routing

!

!

no ip domain-lookup

ip multicast-routing distributed

!

license boot level ipservices

!

spanning-tree mode rapid-pvst

spanning-tree portfast bpduguard default

spanning-tree portfast bpdufilter default

spanning-tree extend system-id

spanning-tree vlan 1-4094 priority 40960

!

!

!

!

vlan internal allocation policy ascending

!

!

!

--More--         !

interface Port-channel10

description Port_Channel_To_C3750

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 151,154,155,160-162

switchport mode trunk

!

interface Port-channel20

description Port_Channel_To_VNXe_3100_SP-A

switchport access vlan 152

switchport mode access

flowcontrol receive on

!

interface Port-channel21

description Port_Channel_To_VNXe_3100_SP-B

switchport access vlan 152

switchport mode access

flowcontrol receive on

!

interface Port-channel37

description Port_Channel_To_Nexus_5Ks

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 140,141,155,156,160-162,180

switchport mode trunk

!

interface FastEthernet0

no ip address

no ip route-cache cef

no ip route-cache

no ip mroute-cache

!

interface GigabitEthernet0/1

description 6248-A_mgmt0

switchport access vlan 154

spanning-tree portfast

!

interface GigabitEthernet0/2

description 5548-A_mgmt0

switchport access vlan 151

--More--          spanning-tree portfast

!

interface GigabitEthernet0/3

description 9148-A_mgmt0

switchport access vlan 151

spanning-tree portfast

!

interface GigabitEthernet0/4

description Vblock_Storage_Mgmt_A

switchport access vlan 151

spanning-tree portfast

!

interface GigabitEthernet0/5

description C200A_CIMC

switchport access vlan 151

spanning-tree portfast

!

interface GigabitEthernet0/6

description To_C200A_G1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 151-154,159-162

switchport mode trunk

spanning-tree portfast trunk

!

interface GigabitEthernet0/7

description To_C200A_G2

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 151-154,159-162

switchport mode trunk

spanning-tree portfast trunk

!

interface GigabitEthernet0/8

description VNXe_A

switchport access vlan 151

spanning-tree portfast

!

interface GigabitEthernet0/9

description To_VNXe_3100_SP-A

switchport access vlan 152

--More--          switchport mode access

flowcontrol receive on

spanning-tree portfast

channel-group 20 mode active

!

interface GigabitEthernet0/10

description To_VNXe_3100_SP-A

switchport access vlan 152

switchport mode access

flowcontrol receive on

spanning-tree portfast

channel-group 20 mode active

!

interface GigabitEthernet0/11

description laptop

switchport access vlan 151

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/12

description laptop

switchport access vlan 151

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/13

description 6248-B_mgmt0

switchport access vlan 154

spanning-tree portfast

!

interface GigabitEthernet0/14

description 5548-B_mgmt0

switchport access vlan 151

spanning-tree portfast

!

interface GigabitEthernet0/15

description 9148-B_mgmt0

switchport access vlan 151

spanning-tree portfast

--More--         !

interface GigabitEthernet0/16

description Vblock_Storage_Mgmt_B

switchport access vlan 151

spanning-tree portfast

!

interface GigabitEthernet0/17

!

interface GigabitEthernet0/18

!

interface GigabitEthernet0/19

!

interface GigabitEthernet0/20

description VNXe_B

switchport access vlan 15

spanning-tree portfast

!

interface GigabitEthernet0/21

description To_VNXe_3100_SP-B

switchport access vlan 152

switchport mode access

flowcontrol receive on

spanning-tree portfast

channel-group 21 mode active

!

interface GigabitEthernet0/22

description To_VNXe_3100_SP-B

switchport access vlan 152

switchport mode access

flowcontrol receive on

spanning-tree portfast

channel-group 21 mode active

!

interface GigabitEthernet0/23

switchport access vlan 151

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/24

--More--          description UPLINK TO STATIC 1

no switchport

ip address 172.16.0.22 255.255.255.248

!

interface GigabitEthernet1/1

!

interface GigabitEthernet1/2

!

interface GigabitEthernet1/3

!

interface GigabitEthernet1/4

!

interface TenGigabitEthernet1/1

description To_5548-A_Port_E1/21

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 140,141,155,156,160-162,180

switchport mode trunk

channel-group 37 mode active

!

interface TenGigabitEthernet1/2

description To_5548-B_Port_E1/21

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 140,141,155,156,160-162,180

switchport mode trunk

channel-group 37 mode active

!

interface Vlan1

no ip address

!

interface Vlan140

ip address 172.16.66.1 255.255.255.192

ip pim sparse-dense-mode

!

interface Vlan141

ip address 172.16.66.65 255.255.255.192

ip pim sparse-dense-mode

!

interface Vlan151

description management

--More--          ip address 172.16.64.1 255.255.255.0

!

interface Vlan152

description C200_ESX_Mgmt

ip address 172.16.65.1 255.255.255.0

!

interface Vlan154

description ucs_mgmt_&_kvm

ip address 172.16.67.1 255.255.255.0

!

interface Vlan155

description VBlock ESX Mgt

ip address 172.16.68.1 255.255.255.0

!

interface Vlan156

description VBlock ESX VMotion

ip address 172.16.69.1 255.255.255.0

!

interface Vlan159

description VBlock NFS

ip address 172.16.70.1 255.255.255.0

!

interface Vlan160

description VBlock ESX Build

ip address 172.16.71.1 255.255.255.0

!

interface Vlan161

description VBlock N1K Packet

ip address 172.16.72.1 255.255.255.0

!

interface Vlan162

description VBlock N1K Control

ip address 172.16.73.1 255.255.255.0

!

interface Vlan180

ip address 172.16.80.1 255.255.255.0

!

ip classless

ip route 0.0.0.0 0.0.0.0 172.16.0.21

--More--         !

ip http server

ip http secure-server

!

ip sla enable reaction-alerts

!

!

banner motd ^C

***W A R N I N G***

THIS IS A PRIVATE COMPUTER SYSTEM.

This computer system including all related equipment, network devices,

are provided only for authorized use.

All computer systems may be monitored for all lawful purposes, including

those activities that are authorized for management of the system.

All information including personal information, stored or sent over this

system may be monitored.

Uses of this system, authorized or unauthorized, constitutes consent to

monitoring of this system.

Unauthorized use may subject you to criminal prosecution.

WARNING: Unauthorized access to this system is forbidden and will be

prosecuted by law.

By accessing this system, you agree that your actions may be monitored.

^C

!

line con 0

login local

line vty 0 4

login local

transport input ssh

transport output ssh

line vty 5 15

login

!

ntp clock-period 36026474

ntp server 152.1.226.105 prefer

ntp server 152.2.21.1

end

vb3750stack#  sh ip igmp

% Incomplete command.

vb3750stack#sh ip igmp > int

Vlan140 is up, line protocol is up

  Internet address is 172.16.66.1/26

  IGMP is enabled on interface

  Current IGMP host version is 2

  Current IGMP router version is 2

  IGMP query interval is 60 seconds

  IGMP configured query interval is 60 seconds

  IGMP querier timeout is 120 seconds

  IGMP configured querier timeout is 120 seconds

  IGMP max query response time is 10 seconds

  Last member query count is 2

  Last member query response interval is 1000 ms

  Inbound IGMP access group is not set

  IGMP activity: 2 joins, 0 leaves

  Multicast routing is enabled on interface

  Multicast TTL threshold is 0

  Multicast designated router (DR) is 172.16.66.1 (this system)

  IGMP querying router is 172.16.66.1 (this system)

  Multicast groups joined by this system (number of users):

      224.0.1.40(1)

Vlan141 is up, line protocol is up

  Internet address is 172.16.66.65/26

  IGMP is enabled on interface

  Current IGMP host version is 2

  Current IGMP router version is 2

  IGMP query interval is 60 seconds

  IGMP configured query interval is 60 seconds

  IGMP querier timeout is 120 seconds

  IGMP configured querier timeout is 120 seconds

  IGMP max query response time is 10 seconds

  Last member query count is 2

  Last member query response interval is 1000 ms

  Inbound IGMP access group is not set

  IGMP activity: 1 joins, 1 leaves

  Multicast routing is enabled on interface

  Multicast TTL threshold is 0

  Multicast designated router (DR) is 172.16.66.65 (this system)

  IGMP querying router is 172.16.66.65 (this system)

  No multicast groups joined by this system

vb3750stack#

************************************************************************************************************************

n1k

*************************************************************************************************************************

sh ip igmp snooping querier

Vlan  IP Address       Version   Expires     Port

140   172.16.66.1      v2        00:04:11    port-channel1

141   172.16.66.65     v2        00:04:24    port-channel17

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2012.10.15 11:45:56 =~=~=~=~=~=~=~=~=~=~=~=

!Command: show running-config

!Time: Mon Oct 15 11:32:27 2012

version 4.2(1)SV1(5.1a)

no feature telnet

feature segmentation

banner motd #Nexus 1000v Switch#

ip domain-lookup

switchname VSMVblock

vem 3

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000001f

vem 4

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000007e

vem 5

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000006e

vem 6

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000004e

vem 7

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000005e

vem 8

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000001e

vem 9

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000002e

vem 10

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000000e

vem 11

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000003e

vem 12

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000006f

vem 13

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000002f

vem 14

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000007f

--More--

vem 15

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000004f

vem 16

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000005f

vem 17

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000000f

vem 18

  host vmware id 4be8cdc6-08d2-11e2-0000-00000000003f

bridge-domain tenant-tmt

  segment id 5014

  group 239.1.1.1

snmp-server user admin network-admin auth md5 0xf9146e3932193ebe43f5f11c2aa07ba0 priv 0xf9146e3932193ebe43f5f11c2aa07ba0 localizedke

y

snmp-server community alpine group network-operator

vrf context management

  ip route 0.0.0.0/0 172.16.64.1

vlan 1,140-141,151,155-156,160-162,180

vlan 1

vlan 140

  name vxlan-subnet-01

vlan 141

  name vxlan-subnet-02

vlan 151

  name management

vlan 155

  name vblock_esx_mgmt

vlan 156

  name vblock_esx_vmotion

vlan 160

  name vblock_esx_build

vlan 161

  name vblock_n1k_packet

vlan 162

  name vblock_n1k_control

vlan 180

  name vm_network

--More--

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

  vmware port-group

  shutdown

  description Port-group created for Nexus1000V internal usage. Do not use.

  state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

  vmware port-group

  shutdown

  description Port-group created for Nexus1000V internal usage. Do not use.

  state enabled

port-profile type ethernet Uplink

  vmware port-group

  switchport mode trunk

  switchport trunk allowed vlan 140,151,155-156,160-162,180

  mtu 1600

  channel-group auto mode on mac-pinning

  no shutdown

  system vlan 140,151,155-156,161-162

  state enabled

port-profile type vethernet Network_Management

  vmware port-group

  switchport mode access

  switchport access vlan 151

  no shutdown

  system vlan 151

  state enabled

port-profile type vethernet ESXi_Server_Management

  vmware port-group

  switchport mode access

  switchport access vlan 155

  no shutdown

  system vlan 155

  state enabled

port-profile type vethernet vMotion

  vmware port-group

  switchport mode access

--More--

  switchport access vlan 156

  no shutdown

  system vlan 156

  state enabled

port-profile type vethernet Packet

  vmware port-group

  switchport mode access

  switchport access vlan 161

  no shutdown

  system vlan 161

  state enabled

port-profile type vethernet Control

  vmware port-group

  switchport mode access

  switchport access vlan 162

  no shutdown

  system vlan 162

  state enabled

port-profile type vethernet VM_Network

  vmware port-group

  switchport mode access

  switchport access vlan 180

  no shutdown

  state enabled

port-profile type ethernet Uplink_vxlan_subnet_02

  vmware port-group

  switchport mode trunk

  switchport trunk allowed vlan 141,151,155-156,160-162,180

  mtu 1600

  channel-group auto mode on mac-pinning

  no shutdown

  system vlan 141,151,155-156,161-162

  state enabled

port-profile type vethernet vxlan-01-vmknic

  vmware port-group

  switchport mode access

  switchport access vlan 140

  capability vxlan

--More--

  no shutdown

  state enabled

port-profile type vethernet vxlan-02-vmknic

  vmware port-group

  switchport mode access

  switchport access vlan 141

  capability vxlan

  no shutdown

  state enabled

port-profile type vethernet tenant-tmt-vxlan

  vmware port-group

  switchport mode access

  switchport access bridge-domain tenant-tmt

  no shutdown

  state enabled

port-profile type vethernet vxlan-03-vmknic

  vmware port-group

  switchport mode access

  switchport access vlan 160

  capability vxlan

  no shutdown

  state enabled

vdc VSMVblock id 1

  limit-resource vlan minimum 16 maximum 2049

  limit-resource monitor-session minimum 0 maximum 2

  limit-resource vrf minimum 16 maximum 8192

  limit-resource port-channel minimum 0 maximum 768

  limit-resource u4route-mem minimum 1 maximum 1

  limit-resource u6route-mem minimum 1 maximum 1

  limit-resource m4route-mem minimum 58 maximum 58

  limit-resource m6route-mem minimum 8 maximum 8

interface port-channel1

  inherit port-profile Uplink

  vem 3

interface port-channel2

--More--

  inherit port-profile Uplink

  vem 4

interface port-channel3

  inherit port-profile Uplink

  vem 5

interface port-channel4

  inherit port-profile Uplink

  vem 6

interface port-channel5

  inherit port-profile Uplink

  vem 7

interface port-channel6

  inherit port-profile Uplink

  vem 8

interface port-channel7

  inherit port-profile Uplink

  vem 9

interface port-channel8

  inherit port-profile Uplink

  vem 10

interface port-channel9

  inherit port-profile Uplink

  vem 11

interface port-channel10

  inherit port-profile Uplink

  vem 12

interface port-channel11

  inherit port-profile Uplink

  vem 13

--More--

interface port-channel12

  inherit port-profile Uplink

  vem 14

interface port-channel13

  inherit port-profile Uplink

  vem 15

interface port-channel14

  inherit port-profile Uplink

  vem 16

interface port-channel15

  inherit port-profile Uplink

  vem 17

interface port-channel16

  inherit port-profile Uplink

  vem 18

interface port-channel17

  inherit port-profile Uplink_vxlan_subnet_02

  vem 12

interface mgmt0

  ip address 172.16.64.7/24

interface Vethernet1

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 64 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.009F

interface Vethernet2

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 66 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

--More--

  vmware vm mac 0025.B5A0.007F

interface Vethernet3

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 65 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.006F

interface Vethernet4

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 68 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.004F

interface Vethernet5

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 67 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.005F

interface Vethernet6

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 72 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.001F

interface Vethernet7

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 69 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.002F

interface Vethernet8

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 71 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.000F

--More--

interface Vethernet9

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 70 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.003F

interface Vethernet10

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 73 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.01FF

interface Vethernet11

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 77 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.01CF

interface Vethernet12

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 74 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.01DF

interface Vethernet13

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 75 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.01EF

interface Vethernet14

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 76 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.01BF

interface Vethernet15

  inherit port-profile ESXi_Server_Management

--More--

  description VMware VMkernel, vmk0

  vmware dvport 79 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.008F

interface Vethernet16

  inherit port-profile ESXi_Server_Management

  description VMware VMkernel, vmk0

  vmware dvport 78 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0025.B5A0.01AF

interface Vethernet17

  inherit port-profile tenant-tmt-vxlan

  description Test Server 03, Network Adapter 1

  vmware dvport 803 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0050.56AB.2EE9

interface Vethernet18

  inherit port-profile tenant-tmt-vxlan

  description Test Server 01, Network Adapter 1

  vmware dvport 801 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0050.56AB.2EEA

interface Vethernet19

  inherit port-profile tenant-tmt-vxlan

  description Test Server 02, Network Adapter 1

  vmware dvport 800 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0050.56AB.2EEB

interface Vethernet20

  inherit port-profile vxlan-01-vmknic

  description VMware VMkernel, vmk1

  vmware dvport 736 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0050.5671.336A

interface Vethernet21

  inherit port-profile vxlan-02-vmknic

  description VMware VMkernel, vmk1

  vmware dvport 769 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

--More--

  vmware vm mac 0050.5674.9747

interface Vethernet22

  inherit port-profile vMotion

  description VMware VMkernel, vmk2

  vmware dvport 100 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0050.5678.52DB

interface Vethernet23

  inherit port-profile vMotion

  description VMware VMkernel, vmk2

  vmware dvport 101 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0050.567D.8886

interface Vethernet24

  inherit port-profile vxlan-01-vmknic

  description VMware VMkernel, vmk1

  vmware dvport 737 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0050.567B.653C

interface Vethernet25

  inherit port-profile vMotion

  description VMware VMkernel, vmk2

  vmware dvport 102 dvswitch uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36"

  vmware vm mac 0050.5677.5B29

interface Ethernet3/1

  inherit port-profile Uplink

interface Ethernet3/2

  inherit port-profile Uplink

interface Ethernet4/1

  inherit port-profile Uplink

interface Ethernet4/2

  inherit port-profile Uplink

--More--

interface Ethernet5/1

  inherit port-profile Uplink

interface Ethernet5/2

  inherit port-profile Uplink

interface Ethernet6/1

  inherit port-profile Uplink

interface Ethernet6/2

  inherit port-profile Uplink

interface Ethernet7/1

  inherit port-profile Uplink

interface Ethernet7/2

  inherit port-profile Uplink

interface Ethernet8/1

  inherit port-profile Uplink

interface Ethernet8/2

  inherit port-profile Uplink

interface Ethernet9/1

  inherit port-profile Uplink

interface Ethernet9/2

  inherit port-profile Uplink

interface Ethernet10/1

  inherit port-profile Uplink

interface Ethernet10/2

  inherit port-profile Uplink

interface Ethernet11/1

  inherit port-profile Uplink

--More--

interface Ethernet11/2

  inherit port-profile Uplink

interface Ethernet12/1

  inherit port-profile Uplink_vxlan_subnet_02

interface Ethernet12/2

  inherit port-profile Uplink_vxlan_subnet_02

interface Ethernet13/1

  inherit port-profile Uplink

interface Ethernet13/2

  inherit port-profile Uplink

interface Ethernet14/1

  inherit port-profile Uplink

interface Ethernet14/2

  inherit port-profile Uplink

interface Ethernet15/1

  inherit port-profile Uplink

interface Ethernet15/2

  inherit port-profile Uplink

interface Ethernet16/1

  inherit port-profile Uplink

interface Ethernet16/2

  inherit port-profile Uplink

interface Ethernet17/1

  inherit port-profile Uplink

interface Ethernet17/2

--More--

  inherit port-profile Uplink

interface Ethernet18/1

  inherit port-profile Uplink

interface Ethernet18/2

  inherit port-profile Uplink

interface control0

line console

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.5.1a.bin sup-1

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.5.1a.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.5.1a.bin sup-2

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.5.1a.bin sup-2

svs-domain

  domain id 1

  control vlan 162

  packet vlan 161

  svs mode L2 

svs connection VC01

  protocol vmware-vim

  remote ip address 172.16.64.162 port 80

  vmware dvs uuid "aa b8 2b 50 f1 2b 7c cc-e9 4c f1 95 b3 ef d2 36" datacenter-name Vblock

  max-ports 8192

  connect

vsn type vsg global

  tcp state-checks

vnm-policy-agent

  registration-ip 0.0.0.0

  shared-secret **********

  log-level

VEM Commands

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2012.10.15 11:55:13 =~=~=~=~=~=~=~=~=~=~=~=

[A

~ #

~ # vemcmd show port [A  

  LTL   VSM Port  Admin Link  State  PC-LTL  SGID  Vem Port  Type

   17    Eth12/1     UP   UP    F/B*    306     0    vmnic0 

   18    Eth12/2     UP   UP    F/B*    306     1    vmnic1 

   49     Veth10     UP   UP    FWD       0     1      vmk0 

   50     Veth17   DOWN   UP    BLK       0        Test Server 03 ethernet0 

   51     Veth19   DOWN   UP    BLK       0        Test Server 02 ethernet0 

   52     Veth21     UP   UP    FWD       0     0      vmk1  VXLAN

   53     Veth22     UP   UP    FWD       0     0      vmk2 

  306       Po17     UP   UP    F/B*      0                 

* F/B: Port is BLOCKED on some of the vlans.

Please run "vemcmd show port vlans" to see the details.

~ #

~ # vemcmd show port old

  LTL   VSM Port  Admin Link  State  PC-LTL  SGID  Vem Port  Type

   17    Eth12/1     UP   UP    F/B*    306     0    vmnic0 

   18    Eth12/2     UP   UP    F/B*    306     1    vmnic1 

   49     Veth10     UP   UP    FWD       0     1      vmk0 

   50     Veth17   DOWN   UP    BLK       0        Test Server 03 ethernet0 

   51     Veth19   DOWN   UP    BLK       0        Test Server 02 ethernet0 

   52     Veth21     UP   UP    FWD       0     0      vmk1  VXLAN

   53     Veth22     UP   UP    FWD       0     0      vmk2 

  306       Po17     UP   UP    F/B*      0                 

* F/B: Port is BLOCKED on some of the vlans.

Please run "vemcmd show port vlans" to see the details.

~ # vmcmd    emcmd show vxlan interfaces

LTL           IP       Seconds since Last

                       IGMP Query Received

(* Interface on which IGMP Joins are sent)

------------------------------------------

52    172.16.66.67        10         *

~ #

====================================================================================== If my answers have been helpful in any way, please rate accordingly. Check out my blog for useful vSphere & UCS tips: www.vmtrooper.com

Hello Matthew and Abhinav,

Thank you for your attention to my question.

The problem appears to be specific to one of my servers.

I verified by adding another server to the vxlan uplink and vmk portgroups that i created just for the second vlan 141

The VMs can communicate between 140 and 141 while being on the same segment id without any problem.

I have seen mysterious gremlins like this in the past with 1000V, and usually completely removing the host from the DVS and removing the VEM for a fresh add resolves the problem.

I will attempt to do so with this problematic system.Hopefully, that will clear up the problem.

I am running version 4.2(1)SV1(5.1a).  I know there are newer versions released, but I am required to stick with this version as of now.

Best Regards,

Trevor

====================================================================================== If my answers have been helpful in any way, please rate accordingly. Check out my blog for useful vSphere & UCS tips: www.vmtrooper.com
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: