Joseph - The ClusterXL A/A configuration is a variation of the StoneSoft or Rainfinity clustering technologies that have been used to cluster Solaris and other *NIX flavored servers and firewalls for years. (In fact, StoneSoft filed suit against Check Point in Europe 8 or 9 years ago for patent violations, and lost.) These configurations were very common on Check Point clusters running on Solaris from the late 90's forward - and, as you describe, have unicast IP's with a multicast MAC for the VIP. Even from the days of installing these on the brand new (at the time) 2900 series switches you had to do exactly as you state above - static MAC entries (or in some cases port mirrors) so traffic was directed to both active switch ports. In Active/Passive mode Check Point ClusterXL clusters are almost always "plug and play" today - rarely do the switches need anything beyond speed/duplex settings. The VIP assumes the MAC of the physical NIC it is currently bound to, and therefore there are no issues as far as switch config or proxy ARP entries on the gateways. All of these issues have to do with traffic flowing to the VIP and through the firewall, and the ability of the switch to correctly identify which physical switch port(s) the VIP is currently patched in to. This is one of three types of traffic associated with ClusterXL itself. The second is state synchronization, which is accomplished through a crossover cable and therefore not relevant. Even when using a switch state sync is a typical TCP 18181 connection from a unicast IP/unicast MAC on one gateway to the other through a dedicated interface pair.
The challenge described by CJ is not with the traffic flowing to the VIP, however. It is an entirely separate process - Check Point Clustering Protocol (aka CPHA if filtering in WireShark) is essentially the heart beat traffic. Every interface pair within a Check Point cluster continually communicates with its "partner" interface on the other cluster members. If any packet takes over 100ms or shows more than a 5% loss the gateway is forced in to "probing" mode where it falls back to ICMP to determine the state of the other cluster member. Depending on the CPHA timing settings an active gateway will failover to the passive in as quickly as 500ms or so. ClusterXL will fail over the entire gateway to the standby to avoid complications with asynchronous routing.
Out of the box, CCP is configured to use multicast, but it supports broadcast as well. To change this in real time (no restart required) simply issue the command:
cphaconf set_ccp {broadcast/multicast}
At the Ethernet level, CCP traffic will always have a source MAC of the Magic MAC of 00:00:00:00:xx:yy where XX is the “Cluster ID” – something identical on each cluster member but unique from one cluster to another, and YY is the cluster priority (00, 01, etc.) based on the priority levels set on cluster members within Dashboard on the cluster object. The destination MAC will always be the Ethernet broadcast of ff:ff:ff:ff:ff:ff.
At the IP level the source of CCP will always appear as 0.0.0.0. The destination will always be the network address (ie, x.x.x.0).
Similarly in multicast mode you will see the same traffic at the IP level but at the Ethernet level the destination will now be a IPv4 multicast MAC (ie, 01:00:5e:4e:c2:1e).
In a tcpdump with the –w flag opened in WireShark and a filter applied of just “cpha” (without the quotes) you should see a continual stream of traffic with the same source and destination IPs on all packets (0.0.0.0 and network IP), the destination of either a bcast or mcast MAC and the source MAC alternating between 00:00:00:00:xx:00 and 00:00:00:00:xx:01.
Long story short, the problem CJ is describing is a behavior on the 7K where a packet capture taken on the Check Point interface itself (ie, tcpdump –i eth0 –w capture.cap) ONLY shows CPHA traffic from it’s own source MAC and no packets from it’s partner. A tcpdump on the 7K itself will show traffic from both.
As CJ mentioned, a simple NxOS upgrade will fix the issue per:
This one:CSCtl67036 basically pryer to NX-OS 5.1(3) the nexus will discard packets that have a source of 0.0.0.0. Which in broadcast mode is exactly what the CCP heartbeat is.
We bypassed this one.
CSCsx47620 is the bug for the for static multicast MAC address feature but it requires 5.2 code on the 7k
(NOTE:Additional RAM may be required for the 5.2 update)
Also note that Check Point gateways do support IGMP multicast groups, given that you have the correct license. It is a feature of SecurePlatform Professional on the higher end gateways or as a relatively inexpensive upgrade on the lower end boxes or open platforms. For lab purposes you can simply type “pro enable” at the CLI (without the quotes). As of the latest build there is no technical limitation (no license check) so you can enable advanced routing features as needed for testing in a lab. For step by step details on configuring IGMP on SPLAT Pro go to the Check Point support site and search for sk32702.
This can be a frustrating issue to troubleshoot, so hopefully this helps someone avoid the headaches I ran in to.
