MAster Master VRRP

brtims · ‎09-09-2004

Hi. I am troubleshooting an issue that has been intermittent for months. We have 2 nokia firewalls integrated between CISCO switches. On occasion VRRP announcements do not get from the primary to the secondary. at this point the secondary becomes master but his announcemtns do not make it to the primary. so at this point we have both claiming mastership. When this occurs the switch has 2 entries for VMAC and thus all traffic destined for the firewall is blocked. Any ideas what could cause the firewalls to not see each other? STP is implemented so why wasn't it resolved by STP?

Georg Pauwen · ‎09-09-2004

Hello,

when the problem occurs, do you actually have STP issues as well at the same time ? If so, you might want to consider the ´uplinkfast´ feature on the links between your switches.

Is it possible to post the output of:

debug vrrp error

debug vrrp events

debug vrrp state

when the problem occurs ?

Regards,

Georg

brtims · ‎09-10-2004

No stp errors in the syslog. that is what is driving us nuts. We have cisco works ready to automate a bunch of commands when it occurs and these are some of them. I don't understand what could cause the interface to go down between the firewalls. The starnge part is that it occurs on only 1 vlan on the dwitch and one interface on the firewalls. every other one is aok.

Kevin Dorrell · ‎09-10-2004

STP does sound like a candidate. If the convergence time of the STP (typically 30 seconds) is longer than the VRRP hold timer (10 seconds for HSRP, I don't know for VRRP), then the STP convergence can break the VRRP. Are you sure that your syslog would log any topology changes on the vlan in question?

Kevin Dorrell

Luxembourg

brtims · ‎09-10-2004

I didn't see anything at all. We are on log level 5 (out of 7) The mastership changes almost every other day but only rarely (every couple of weeks) both are in Master state and remain there until the switch port is disabled/enabled. that clears the cam entry (where there were two entries for the vmac) and everything works fine again. It is driving me nuts.

Somewhere on the switch (I suspect) the path for the vrrp announcements get blocked in both directions to/from the two firewalls running vrrp

Kevin Dorrell · ‎09-10-2004

Sorry, did you say that when this happens there were two entries for the VMAC? On the same switch? That would confuse it. How can that come about? I take it they are on the same VLAN?.

Are your firewalls using virtual interfaces to trunk into the switch, or is each firewall interface handled separately?

Kevin Dorrell

Luxembourg

brtims · ‎09-10-2004

yes 2 entries for vmac when the outage occurs on the same switch for that port of in this case 9/15. And yes it all on the same vlan. The firewalls are trucked to the switch using vmac. I know that because when I do a show cam I see the vmac as an entry. However I did notice that all packets coming from the firewall uses the physical as the source MSC insteadof the VMAC. I thought that could be the problem but whne I looked at other vlans and other firewall interfaces they do that as well and there is no outage there

Kevin Dorrell · ‎09-10-2004

I can't get my head around this. How can the switch have two entries in the CAM table for the same MAC address on two different ports on the same VLAN? One is supposed to cancel the other. Unless they are multicast MACs, which would be strange for a firewall, wouldn't it? Maybe it's something about VRRP that I don't know yet; can anyone enlighten me? Maybe I'm assuming it is too much like HSRP.

I don't suppose one of them is a static CAM entry, for example if you have configured port security on the port?

Also, it seems wrong for them to source from the physical addresses. Are you saying they respond to ARP with the VMAC, but source their packets with their physical MAC? What happens when the VMAC CAM entry times out then?

I'm going away to read about VRRP!

Kevin Dorrell

Luxembourg

brtims · ‎09-10-2004

I agree it is wierd. All packets coming from the firewalls are sourced with physical MAC either primary or secondary depending on who is master. all packets coming to the firewalls are destined to the VMAC so the firewall responds to ARP for the VMAC. the outage is for all traffic destined to the VMAC. The traffic flows ok from the firewall to the F5. The problem seems to be that the firewalls cannot talk to each other so neither ones hears the others VRRP announcement so they both claim mastership. Ths for some reason confuses the switch but not every time. Like I mentioned before sometimes they are both master but traffic flows. Other times traffic does not flow. The root cause I am trying to find is why can't the two firewalls talk to each other and when that happens why is it that sometimes traffic still flows to the VMAC and other times it doesn't? I was told that when it doesn't work the show cam dynamic shows two entries for the VMAC so to fix it the disable/enable the port. I have setup scripts to capture that show cam dynamic when it happens because I did not think that could happen at all.

Thoughts?

Kevin Dorrell · ‎09-10-2004

OK, I'm reading http://www.cisco.com/warp/customer/117/vip_appguide.html. I see that VRRP packets are sourced from a 00-00-5e-00-01-(VRID), and multicasted to (what address?) Only the primary generates the VRRP packets; the other just listens (except in your case of course ;-) Are you saying you see two entries for 00-00-5e-00-01-(VRID)? I suppose the destination multicast must be one of those that is unaffected by cgmp or igmp snooping, something like 01-00-5e-00-01-(VRID) perhaps.

Keep reading Kevin!

Kevin Dorrell

Luxembourg

brtims · ‎09-10-2004

OUR VRID or VMAC is 00-00-5e-00-01-9a. the multicast address is 01:00:5e:00:00:12. Normally I see this in the cam:

VLAN 111

show cam dynamic 111

* = Static Entry. + = Permanent Entry. # = System Entry. R = Router Entry.

X = Port Security Entry $ = Dot1x Security Entry

VLAN Dest MAC/Route Des [CoS] Destination Ports or VCs / [Protocol Type]

---- ------------------ ----- -------------------------------------------

111 00-00-5e-00-01-9a 1/1-2,2/1-2 [ALL]

Total Matching CAM Entries Displayed =1

From what I hear there are two entries for this when it occurs. I want to verify myself.

Kevin Dorrell · ‎09-10-2004

Is that a 4-way channel interface I see there as a destination port? If so, are you sure that the channel is stable? Is the channel dynamically negotiated? When you see the two entries, is it a completely different interface, or maybe is it fragments of the channel group? It will be interesting to see.

Kevin Dorrell

Luxembourg

brtims · ‎09-10-2004

I am really thinking that VRRP is aok. It does change mastership once and a while probably due to the STP convergence which is normal if the VRRP timeout is less than the convergence time. I think somewhere along the line there is some STP topology that screws up the path from primary firewall to secondary firewall. what this could be is what I am looking for ideas on. Could there be some STP topology that screws up the whole thing or could two nodes claiming the same VMAC cause STP to be confused?

Kevin Dorrell · ‎09-10-2004

Does PC1 see any traffic at all from VLAN 4, e.g. flooded unicasts? Or is it just the ARP that gets lost? There are a couple of bugs in the 4000 where the span destination port shifts into the wrong VLAN. CSCdt07747 and CSCdt13708. I know its a different model of Catalyst, but I'm clutching at straws here. The latter bug suggests you try putting the destination port back into vlan 4 after you have set up the span session.

Kevin Dorrell

Luxembourg

Kevin Dorrell · ‎09-10-2004

No, I still don't get it. VRRP looks very similar to HSRP, so no surprises there. The switch's behaviour is bizarre. You don't have port security on the ports or anything like that do you? I'll get back to you if anything occurs to me over the weekend.

Kevin Dorrell

Luxembourg