Nexus 1000v Removing VEM # heartbeats lost

[Indika Wasala]

Dear All,

We have deployed a Nexus 1000v and when we add the second uplink interface of the ESXi server to Uplink port group it says, "Removing VEM 3 (hearbeats lost) and it removes the vem. When you do show modules none of the modules appear after adding the second interface. We use L2 packet/control. We are connecting the VSMs to standard switch and in all ESXi uplink interfaces packet/control vlans are allowed.

When we check hosts under Nexus 1k switch in vcenter it shows all hosts as connected and VDS status as UP.

VEM version - cisco-vem-v160-esx             4.2.1.2.2.1.0-3.1.1

ESXi 5.1 U1

Appreciate your help.

Thanks,

Best Regards,

Indika.

[sprasath]

Hi Indika,

- Are you using the same uplink port-profile for your second uplink as well?

- What type of servers are these (UCS/HP/Dell etc)?

- What is the control VLAN that you are using?

- Paste the configuration of your uplink port-profile(s)

- What is connected to the servers (switch etc) and what is the configuration on these ports?

- Does the VEM stay connected with the VSM if you use only the second uplink (and not both the first and second)?

Thanks,

Shankar

[Indika Wasala]

Hi Shankar,

- We are using the same uplink port profile for the second uplink as well.

- Servers are UCS C240 managed by UCSM (6248UP FIs)

- We use L2 VLAN as control/packet VLAN (824).

- Below is the uplink port profile config,

port-profile type ethernet Uplink

  vmware port-group

  switchport trunk allowed vlan 106,400-401,822,824

  switchport mode trunk

  channel-group auto mode on mac-pinning

  no shutdown

  system vlan 822,824

  state enabled

- Hardware config is as in the image

each server is having one 1225vic which connects to both N2K. N2ks in-turn connect to FI-A and B respectively.

- If we use only the second uplink, it doen't work at all.. (VEM does not appear at all..)

Interestingly when I bring down any data path (Either LAN uplinks from one FI to N5ks or server ports connected to N2k) in FI, VSM detects VEMs. But when both paths are connected it removes VEMs saying heartbeat lost.

Thanks,

Indika.

[sprasath]

With just the second uplink, you've mentioned that it doesn't work at all. Does it work with just the second uplink if you bring down any data path?

If yes, what exactly are you bringing down?

And with two uplinks, have you tried all 4 options and confirm that it still works (i.e bring down FI A/FI B uplink, bring down server ports connecting to either N2k)

If all these are true, then it's most likely that your control VLAN is not allowed (or blocked) in your upstream N5k.

Thanks,

Shankar

[Indika Wasala]

Hi Shankar,

Thanks for the reply,

Actually we didn't bring down uplinks/server ports to one N2k with only second uplink connected. We will try it tomorrow.

We tried brining down FI-A LAN uplinks with both uplinks connected and it worked fine (VEMs were appearing). It also works (2 uplinks connected) fine when we bring down one side N2k server ports as well.

Thanks,

Indika.

[sprasath · Accepted solution]

To me, it looks like an upstream issue either way. So please make sure that the VLAN is allowed and not blocked on the upstream switches.

Thanks,

Shankar

[Indika Wasala]

Thanks Shankar,

Will check tomorrow and post.

[Indika Wasala]

Thanks a lot Shankar,

It was due to missing pkt-ctrl vlan in upstrem N5Ks.