07-12-2009 12:29 AM
Hi,
i'm going to use Nexus 7000 in Data Center.
During analysis configuration, I need define mac-address-static configuration for multicast mac address for Firewall Checkpoint cluster.
In "Layer 2 Switching Configuration Guide, Release 4.1.pdf" documentation speak about
"Configuring a Static MAC Address
[..]You cannot configure broadcast or multicast addresses as static MAC addresses[..]"
Have you a suggestion to manage this problem and why is it not possible configure mac address static multicast?
Regards
Dino
09-26-2011 07:29 PM
Joseph did you ever get this to work with the latest NX-OS ?
09-27-2011 05:53 AM
Yes there is a fix or a workaround for this. What we did was to have the Checkpoint Firewall host the multicast MAC and respond using proxy-arp for the VIP. We also turned off igmp snooping on the Nexus 7k. For a layer 3 interface on the switch the 'ip arp' command worked to allow the switch to communicate to the VIP of the firewall.
The other options would be to turn on broadcast HA on the checkpoints but that would be a less desireable setup in our situation - or - in later releases of the checkpoint software you can actually setup the firewall to join an igmp group.
I hope this helps.
09-27-2011 10:15 PM
what we are seeing now that the checkpoint guy has explained it is the CPP traffic is never making it to each of the cluster members when in the 7k. If you move the interface to a 3750 the cpp packets work. It doesn't matter if its in mulitcast or broadcast mode with IGMP snooping disabled. Any ideas ?
Cisco says upgrade the the 7k to 5.2 code so we can do a static mac entry with the multicast address of the CCP packet source.
09-29-2011 06:09 AM
We are only running 5.1(1a).
You must disable IGMP snooping. It sounds like you did.
If you don't have a layer 3 interface on the 7K for the subnet then you must enable the checkpoint firewall to advertise the arp entry for that subnet.
If you are routing through the 7K for the end host then all you should need is the static MAC entry for the VIP of the checkpoint cluster to allow the traffic to be routed correctly.
This all stems from how Checkpoint accomplishes its active/active load balancing. In multicast mode (which in our lab when we first tested checkpoint worked the best), the checkpoint cluster IP is a unicast IP address and is associated with a multicast MAC address. You must disable igmp snooping for that subnet to allow the mulicast packets to be treated as a broadcast by the switch (forwarded out all interfaces in that VLAN except for the interface that it was recieved on). When the end host sends a packet destine for another host on a different network and the checkpoint firewall is next hop/Defualt gateway, an arp request has to be generated by the end host asking the MAC of the DG (and in checkponts case is a multicast MAC). The fix for this if you have the 7K routing to the firewall is to statically set the MAC in the layer 3 interface configuration, like so:
interface Vlan10
no shutdown
ip address 192.168.100.1/24
ip arp 192.168.100.10 0100.5e11.1111
As for the CCP packets between the checkpoints, we never had any issues with those packets traversing the 7K. Make sure that the checkpoint firewalls are running proxy-arp as stated in my previous post.
I hope this helps!
10-11-2011 09:36 AM
Thanks for the reply. So what actually happened. We didn't have an active/active Checkpoint cluster we had an active/passive. This checkpoint was the eggress gateway for the Nexus but also plugged into the nexus core VDC via 1 Gig. After working with TAC and doing packet captures we saw the CCP packets both in Broadcast mode and Mulitcast mode packets. The checkpoint vendor basically show us where the cluster was flipping back and forth on the nexus because it was not recieve the CCP heartbeats on that interface. TAC found us two bugs and we able to fix one of them.
This one:
CSCtl67036 basically pryer to NX-OS 5.1(3) the nexus will discard packets that have a source of 0.0.0.0. Which in broadcast mode is exactly what the CCP heartbeat is.
We bypassed this one.
CSCsx47620 is the bug for the for static multicast MAC address feature but it requires 5.2 code on the 7k and that then requires.
10-11-2011 03:25 PM
Joseph - The ClusterXL A/A configuration is a variation of the StoneSoft or Rainfinity clustering technologies that have been used to cluster Solaris and other *NIX flavored servers and firewalls for years. (In fact, StoneSoft filed suit against Check Point in Europe 8 or 9 years ago for patent violations, and lost.) These configurations were very common on Check Point clusters running on Solaris from the late 90's forward - and, as you describe, have unicast IP's with a multicast MAC for the VIP. Even from the days of installing these on the brand new (at the time) 2900 series switches you had to do exactly as you state above - static MAC entries (or in some cases port mirrors) so traffic was directed to both active switch ports. In Active/Passive mode Check Point ClusterXL clusters are almost always "plug and play" today - rarely do the switches need anything beyond speed/duplex settings. The VIP assumes the MAC of the physical NIC it is currently bound to, and therefore there are no issues as far as switch config or proxy ARP entries on the gateways. All of these issues have to do with traffic flowing to the VIP and through the firewall, and the ability of the switch to correctly identify which physical switch port(s) the VIP is currently patched in to. This is one of three types of traffic associated with ClusterXL itself. The second is state synchronization, which is accomplished through a crossover cable and therefore not relevant. Even when using a switch state sync is a typical TCP 18181 connection from a unicast IP/unicast MAC on one gateway to the other through a dedicated interface pair.
The challenge described by CJ is not with the traffic flowing to the VIP, however. It is an entirely separate process - Check Point Clustering Protocol (aka CPHA if filtering in WireShark) is essentially the heart beat traffic. Every interface pair within a Check Point cluster continually communicates with its "partner" interface on the other cluster members. If any packet takes over 100ms or shows more than a 5% loss the gateway is forced in to "probing" mode where it falls back to ICMP to determine the state of the other cluster member. Depending on the CPHA timing settings an active gateway will failover to the passive in as quickly as 500ms or so. ClusterXL will fail over the entire gateway to the standby to avoid complications with asynchronous routing.
Out of the box, CCP is configured to use multicast, but it supports broadcast as well. To change this in real time (no restart required) simply issue the command:
cphaconf set_ccp {broadcast/multicast}
At the Ethernet level, CCP traffic will always have a source MAC of the Magic MAC of 00:00:00:00:xx:yy where XX is the “Cluster ID” – something identical on each cluster member but unique from one cluster to another, and YY is the cluster priority (00, 01, etc.) based on the priority levels set on cluster members within Dashboard on the cluster object. The destination MAC will always be the Ethernet broadcast of ff:ff:ff:ff:ff:ff.
At the IP level the source of CCP will always appear as 0.0.0.0. The destination will always be the network address (ie, x.x.x.0).
Similarly in multicast mode you will see the same traffic at the IP level but at the Ethernet level the destination will now be a IPv4 multicast MAC (ie, 01:00:5e:4e:c2:1e).
In a tcpdump with the –w flag opened in WireShark and a filter applied of just “cpha” (without the quotes) you should see a continual stream of traffic with the same source and destination IPs on all packets (0.0.0.0 and network IP), the destination of either a bcast or mcast MAC and the source MAC alternating between 00:00:00:00:xx:00 and 00:00:00:00:xx:01.
Long story short, the problem CJ is describing is a behavior on the 7K where a packet capture taken on the Check Point interface itself (ie, tcpdump –i eth0 –w capture.cap) ONLY shows CPHA traffic from it’s own source MAC and no packets from it’s partner. A tcpdump on the 7K itself will show traffic from both.
As CJ mentioned, a simple NxOS upgrade will fix the issue per:
This one:CSCtl67036 basically pryer to NX-OS 5.1(3) the nexus will discard packets that have a source of 0.0.0.0. Which in broadcast mode is exactly what the CCP heartbeat is.
We bypassed this one.
CSCsx47620 is the bug for the for static multicast MAC address feature but it requires 5.2 code on the 7k
(NOTE:Additional RAM may be required for the 5.2 update)
Also note that Check Point gateways do support IGMP multicast groups, given that you have the correct license. It is a feature of SecurePlatform Professional on the higher end gateways or as a relatively inexpensive upgrade on the lower end boxes or open platforms. For lab purposes you can simply type “pro enable” at the CLI (without the quotes). As of the latest build there is no technical limitation (no license check) so you can enable advanced routing features as needed for testing in a lab. For step by step details on configuring IGMP on SPLAT Pro go to the Check Point support site and search for sk32702.
This can be a frustrating issue to troubleshoot, so hopefully this helps someone avoid the headaches I ran in to.
05-28-2014 06:17 PM
Good Night All,
I am presently having this issue with a Checkpoint cluster operating in load sharing multicast mode connected to a nexus 5500 series cluster core.
When in this mode all switches including the nexus are not able to arp to the VIP of the respective firewall interfaces. So what is occurring is that the DG of the nexus as well as 3com switches in the customer's environment is having an incomplete ARP entry for the Firewall VIP.
I have seen in stated below that RFC 1812 does not allow this. Can you advise if this is the reason why all switches with a layer 3 interface are not able to arp to the respective DG on the Firewall cluster?
Are there any workaround for this beside entering manually arp entries on the switches to point to the multicast mac?
We are trying to explain to the customer that this is the way forward however they are of the belief that the core switch (nexus) is the issue and once resolved there will allow the other switches 3com to communicate without the need of the multicast mac static entry.
Please help!!! I have seen much discussions here and we presently have a cisco tac case opened but no real root cause determined by cisco.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide