Re: n1k and VRRP

m.sekiguchi · ‎03-29-2011

Hi,

I'm now trying to run the VRRP on the virtual appliance with cisco nexus 1000v.

However, I've not succeeded in making the failover work.

As written in the deployment guide, VEM has the simple mechanism to detect the possible loop.

Since VRRP uses the same VR-MAC address, it seems that the VRRP advertisement packet is not forwarded to the virtual machine running on another VEM.

When the failover occurs, the backup device starts sending the VRRP advertisement (purple line).

As a result, VEM#2 learns the VR-MAC address, 00:00:5e:00:01:0C, as a local MAC address.

As you may notice, VEM#1 has already learned the same MAC address as a local MAC address.

Thus the VRRP advertisement from the backup VM cannot reach the active device.

After the active device comes back online, it simply starts sending the VRRP advertisement (orange line).

Again, this VRRP advertisement does not reach the backup device since VR-MAC already exists in the local mac address table.

Does anybody know how to configure nexus 1000v to support VRRP on the connected virtual appliance?

Can I disable the loop prevention mechanisim on VEM?

ESXi: Ver. 4.1

Nexus 1000v: 4.2(1)SV1(4)

Regards,

Masato

stmccabe · ‎04-04-2011

Hello Masato,

By default the multicast should get sent out all L2 connected links, unless IGMP snooping is enabled. Do you have igmp snooping enabled? What is the configuration on the virtual appliance (e.g., checkpoint vFW).

Thanks

Stephen McCabe

m.sekiguchi · ‎04-04-2011

Hello Stephen,

Thanks for your reply.

I disabled IGMP snooping globaly; no ip igmp snooping.

Sending out the VRRP multicast packet from virtual appliances is fine.

And receiving the VRRP multicast packet sent from another network node (L3 switch, routers, load-balancers) is also fine.

The problem is receiving the VRRP multicast packet sent from the virtual appliance on other ESXi host.

Here is the example of MAC address table of VEM.

DVS-Nexus# show mac address-table module 3 vlan 201
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---

201 0000.5e00.010c dynamic 1 Eth3/3 3

201 0003.b280.0040 dynamic 1 Eth3/3 3 (VRRP master)

DVS-Nexus# show mac address-table module 4 vlan 201
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---

201 0000.5e00.010c dynamic 1 Veth16 4

201 0003.b280.0040 dynamic 1 Veth16 4 (VRRP Master)

The virtual appliance on the VEM module #4 is now active. It's virtual mac address is 00:03:b2:80:00:40 and the VRRP VR-MAC is 00:00:5e:00:01:0c.

Since this VR-MAC address is not recorded as a local MAC entry on the VEM module #3, all nodes on the vlan 201 on the VEM module #3 can receive the VRRP advertisement from 00:03:b2:80:00:40.

Now, I boot the backup device on the VEM module4.

DVS-Nexus# show mac address-table module 3 vlan 201
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---

201 0000.5e00.010c dynamic 1 Eth3/3 3

201 0003.b280.0040 dynamic 1 Eth3/3 3 (VRRP master)

201 0003.b280.9b80 dynamic 0 Veth3 3 (VRRP backup)

DVS-Nexus# show mac address-table module 4 vlan 201
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---

201 0000.5e00.010c dynamic 1 Veth16 4

201 0003.b280.0040 dynamic 1 Veth16 4 (VRRP Master)

201 0003.b280.9b80 dynamic 2 Eth4/2 4 (VRRP backup)

As you can see, the backup device, 00:03:b2:80:9b:80, is now connected to Veth3 on the VEM#2 (module 3).

Since VR-MAC, 00:00:5e:00:01:0c, is now learned as a external node on VEM#1, the backup device can receive the VRRP advertisement from the active device, 00:03:b2:80:00:40. The VRRP works fine at this moment.

Next, I shutdown the active unit.

Since the backup device does not receive the VRRP advertisement, it sends the gratuitous ARP and starts sending the VRRP advertisement.

As a result, the mac address table becomes as follows;

DVS-Nexus# show mac address-table module 3 vlan 201
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---

201 0000.5e00.010c dynamic 1 Veth3 3

201 0003.b280.9b80 dynamic 0 Veth3 3 (becomes Master)

DVS-Nexus# show mac address-table module 4 vlan 201
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---

201 0000.5e00.010c dynamic 0 Eth4/2 4

201 0003.b280.9b80 dynamic 0 Eth4/2 4

As you can see, the VR-MAC, 00:00:5e:00:01:0c, now becomes local MAC address entry of the VEM module #3.

Then I restart the active device, 00:03:b2:80:00:40, on the VEM module #4.

Since 00:03:b2:80:00:40 has the higher priority than 00:03:b2:80:9b:80, it becomes the master device and starts sending the VRRP advertisement.

Here is the mac address table after booting 00:03:b2:80:00:40.

DVS-Nexus# show mac address-table module 3 vlan 201
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---

201 0000.5e00.010c dynamic 1 Veth3 3

201 0003.b280.0040 dynamic 0 Eth3/3 3

201 0003.b280.9b80 dynamic 0 Veth3 3

DVS-Nexus# show mac address-table module 4 vlan 201
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---

201 0000.5e00.010c dynamic 1 Veth16 4

201 0003.b280.0040 dynamic 0 Veth16 4 (VRRP master)

201 0003.b280.9b80 dynamic 0 Eth4/2 4

As you can see, both VEMs learn 00:00:5e:00:01:0c as a local mac address.

This prohibits the nodes on the VEM module #3 to receive the VRRP advertisement from VEM module#4 and vice versa.

As a result, both 00:03:b2:80:00:40 and 00:03:b2:80:9b:80 become master.

The current workaround is to disable the preemption on the virtual appliance, but I want to know if there is any way to disable or tune the loop prevention mechanis on VEM.

Regards,

Masato

stmccabe · ‎04-05-2011

Masato, 

If not configured you will need to add the inter firewall HA vlan as N1k system vlan on the appropriate port profiles for the vFWs.  
This should allow the multicast to be sent between FW and other ESXi host. This allows L2 forwarding to occur without the VSM intervening.

For example, let's say the HA VLAN is 200:

port-profile type ethernet SYSUPLINK
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-3967,4048-4093
no shutdown
system vlan 200,509-512
state enabled

Now inorder to change this you will need to shut down the sysuplink, so you may need to create another system uplink (e.g., SYSUPLINK2), and then migrate your connections to your vSwitch then over to new sysuplink..

m.sekiguchi · ‎04-10-2011

Dear Stephen,

Thanks. The virtual appliance I'm now testing is the virtual load-balancer.

I changed VLANs to the system vlan, but the result was same as before.

port-profile type ethernet system-uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 100-101,201-202,301,401,900

no shutdown

system vlan 100-101,201-202,301,401,900

state enabled

port-profile type vethernet VA_Client

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 100,401

no shutdown

system vlan 100,401

state enabled

port-profile type vethernet VA_Server

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 201-202

no shutdown

system vlan 201-202

state enabled

My virtual appliance uses VLAN 100, 201-202, and 401.

As you can see in the following mac address table, each VEM learns the VR-MAC, 00:00:5e:00:01:29, as a local node and does not forward VRRP advertisements received by the physical port (system-uplink).

DVS-Nexus# show mac address-table module 3 vlan 100

VLAN MAC Address Type Age Port Mod

---------+-----------------+-------+---------+------------------------------+---

100 000c.2996.ce85 static 0 Veth2 3

100 0000.5e00.010a dynamic 0 Eth3/2 3

100 0000.5e00.0129 dynamic 0 Veth2 3

100 0001.30bb.38f0 dynamic 1 Eth3/2 3

100 0001.30bb.54b0 dynamic 32 Eth3/2 3

100 0003.b280.9b80 dynamic 1 Veth2 3

DVS-Nexus# show mac address-table module 4 vlan 100

VLAN MAC Address Type Age Port Mod

---------+-----------------+-------+---------+------------------------------+---

100 0050.56ac.0003 static 0 Veth15 4

100 0000.5e00.010a dynamic 1 Eth4/3 4

100 0000.5e00.0129 dynamic 1 Veth15 4

100 0001.30bb.38f0 dynamic 1 Eth4/3 4

100 0001.30bb.54b0 dynamic 35 Eth4/3 4

100 0003.b280.0040 dynamic 1 Veth15 4

I think the loop prevention mechanism is running on each VEM without any intervention of VSM.

That's why it can prevent the loop without STP.

Can we conclude that the HA solution which use the same virtual MAC address like VRRP does not work on n1k due to VEM loop prevention mechanism?

Regards,

Masato

stmccabe · ‎04-11-2011

Hi Masato,

Can you tell me what virtaul firewall product you are leveraging? VRRP should work and I have used VRRP in my own setup. Please let me know what ova you are testing.

Thanks.

m.sekiguchi · ‎04-12-2011

Dear Stephen,

I'm evaluating the virtual load-balancer. Not a firewall product.

Since this is an experimental build, I need to find the root cause of this failure in the VRRP failover.

As far as I took a capture on the virtual machine, I think VEM blocks the VRRP advertisement.

I've not tested, but I guess VRRP will work if I run both master/backup virtual appliances on the same VEM (same ESXi host).

Do you run your virtual firewall pairs on the different VEM (ESX/ESXi host)?

Did you check the similar fail-over scenario as mine in your lab? My scenario is;

preemption = enable
- Instance with higher priority is always VRRP master.
reboot the master virtual firewall which has higher priority than the backup vFW
- This causes the backup device becomes the master, and it will be back to the backup status once the pre-master vFW becomes active.
  - In my setup, both device became the master status.

It would be highly appreciated if you can share the mac address table info during the VRRP failover and the details in the VRRP packet.

If VRRP works exactly in your environment with my failover scenario, I wonder what kind of trick the VEM loop prevention mechanism is using.

Just in case, here is the screen shot of VRRP packet my virtual load-balancers are sending.

[VRRP master: 172.16.0.41]

[VRRP backup: 172.16.0.42 - this VRRP advertisment is sent while 172.16.0.41 is down]

The failover due to the down of the virtual appliance works fine.

I think this is because VEM can clear learned MAC address from local mac address table.

Regards,

Masato

malguy_rentabiliweb · ‎04-20-2011

Hello,

I am also very interested in this topic. Instead of VRRP I'm testing with CARP (OpenBSD, for firewalling stuff) which this is very very close to VRRP in term of protocol details (same Virtual MAC set notably).

I've been able to make CARP work with the original Virtual Standard Switch and the Virtual Distributed Switch only and only if I accept the interfaces to be in promiscious mode, which is unacceptable to me (security and load issues), which is the reason I've decided to evaluate N1K.

I've tried adding the VLANs where CARP is present to both my Ethernet and vEthernet port-profiles as system VLANs without success.

In my lab setup, I have two ESX hosts. Each runs a virtualized FW + a virtualized monitoring station + a VEM. One has a VSM + a vCenter. The two ESX hosts share the same upstream switch (a Cisco 2960G). I also have a third monitoring station (not virtualized at all) connected to the upstream switch, with the port in trunk mode.

I obtain the exact same results as Masato (outside VRRP/CARP announces received on the vFWs , vFWs' CARP announces received outside the ESX hosts).

So, at this point, I'm pretty stuck.

sbhadrav@cisco.com · ‎02-16-2018

The VRRP virtual MAC address (or virtual router MAC address) is a shared MAC address adopted by the VRRP master. If the master fails the same virtual MAC master fails over to the new master. As a result, all packets for VRRP routers can continue to use the same virtual MAC address. You must enable the VRRP virtual MAC address feature on all members of a VRRP group.

Each VRRP router is associated with its own virtual MAC address. The last part of the virtual MAC depends on the VRRP virtual router ID using the following format:

00-00-5E-00-01-<VRID_hex>

Where <VRID_hex> is the VRRP virtual router ID in hexadecimal format in internet standard bit-order. For more information about the format of the virtual MAC see RFC 3768.
If the VRRP virtual router ID is 10 the virtual MAC would be 00-00-5E-00-01-0a.
If the VRRP virtual router ID is 200 the virtual MAC would be 00-00-5E-00-01-c8.
The VRRP virtual MAC address feature is disabled by default. Wen you enable the feature on a FortiGate interface, all of the VRRP routers added to that interface use their own VRRP virtual MAC address. Each virtual MAC address will be different because each virtual router has its own ID.
Use the following command to enable the VRRP virtual MAC address on the port2 interface:
config system interface

edit port2

set vrrp-virtual-mac enable

end

The port2 interface will now accept packets sent to the MAC addresses of the VRRP virtual routers added to this interface.

Using the VRRP virtual MAC address can improve network efficiency especially on large and complex LANs because when a failover occurs devices on the LAN do not have to learn a new MAC address for the new VRRP router.

If the VRRP virtual MAC address feature is disabled, the VRRP group uses the MAC address of the master. In the case of a FortiGate VRRP virtual router this is the MAC address of the FortiGate interface that the VRRP virtual routers are added to. If a master fails, when the new master takes over it sends gratuitous ARPs to associate the VRRP virtual router IP address with the MAC address of the new master (or the interface of the FortiGate that has become the new master). If the VRRP virtual MAC address is enabled the new master uses the same MAC address as the old master.