cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
700
Views
1
Helpful
8
Replies

Ping drop

grapevine
Level 5
Level 5

We have a stack of 3 cisco switches in stack connected to Palo Alto in HA in LACP etherchannel
Firewall 1 has links to Gi1/0/23 and Gi3/0/11
Firewall 2 has links to Gi2/0/23 and Gi3/0/13
We have lacp rate as slow and configured lacp fast-switchover on interface portchannels
We did a failover testing and when switch 3 was powered off firewall is bringing all portchannel-links down on the active firewall and failing over to passive
this is causing a lot of ping drops
When switch 2 and switch 1 are powered off when active, we see only one ping drop
We shut the ports Gi3/0/11 and Gi3/0/13 one by one and did the failover testing still the same results
We configured lacp rate fast and spanning tree portfast trunk on the port channel still the same issue
Please help us fix this

 

8 Replies 8

mloraditch
Meraki Community All-Star
Meraki Community All-Star

Do the Palo Alto logs show anything different between when SW3 is removed vs SW1 or SW2? Based on your topology, removing SW3 kills a link on both PAs at once. I suspect something is renegotiating in that instance that doesn't happen when you just pull SW1 or SW2.

If you found this post helpful, please give it a thumbs up. If my answer solves your problem please click Accept as Solution so others can benefit from it.

Joseph W. Doherty
Hall of Fame
Hall of Fame

Is SW3 the active master?

Is your stack also configured not to change the stack's MAC if there is a stack master failure?

BTW, it's often helpful if you identify the specific switch models and the IOS version being used.

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello @grapevine ,

first of all check who is the master in the stack with

show switch

as suggested you should implement MAC address persistency so that also in case of change of master there are no impacts also in STP.

Hope to help

Giuseppe

 

@grapevine as you haven't answered my questions, cannot with high assurance answer your questions.  However, what I suspect (likely so does @Giuseppe Larosa ) there can be a noticeable difference in operational impact between a non stack member failing vs. the stack master failing.  The latter, i.e. a stack master failure, can be more impactful, as it's the operational brains of the stack.

One issue, on many stacks, if the stack master fails, by default, the stack uses a new MAC for stack operation.  However, there's often a configuration option to retain the original stack MAC even when the stack master fails.  This often mitigate the impact of a stack master failure.

BTW, if the stack is being used for L3 operation, there are are often (?) other configuration options to minimize the impact of a stack master failure for those too.

Again, though, as you didn't answer my questions, cannot say with much certainty the cause of the behavior you note, but I can say, from experience, loss of a stack master can be more operationally impactful, especially if you've not configured the options to mitigate its impact. 

I also see both links in portchannel on firewall going down and causing firewall failover as well

@grapevine , again, it can be very helpful when you provide additional base information, such as the actual physical devices being used and the IOS version being used.  (Sometimes it helps others find a documented bug that covers the issue you're encountering.). Further, answering any specific requests for specific information.

So far, you've confirmed the ping loss, and drops of Etherchannel links, on FW, when stack master is powered off (?).

From what you describe, the impact of loss of stack master seems more impactful than I would have expected.  Assuming a bug is not involved, I would suspect, possibly, the configuration isn't optimal.  I have in mind SSO (I believe this is mostly/fully "automatic" for a stack), NSF (often needed for L3 protocols, like routing, to keep neighbor relationships active during fail over recovery - also may require neighbors to be NSF aware [FW?]) and possibly LACP parameters too.

From your description of ping drops, you didn't describe whether ping loss was for stack as source or destination, or all ping traffic was transit, or different results for transit traffic vs. local src/dst.

The big issue appears to be Etherchannel links seen going down on FW.  I've no experience with a Palo Alto FW, and possibly, it and a Cisco physical stack might not ever play nicely together if certain physical stack member failures.  Possibly, the FW needs configuration changes.

Assuming the PA FW and Cisco physical stack cannot cooperate, one or both, unfortunately, might need to be replaced.  On the Cisco side, possibly a Stackwise Virtual might work with the FW.

Have you contacted either/both Palo Alto and/or Cisco TAC?  What you're trying to accomplish isn't oddball and both vendors should have more experience with this kind of requirement.

sw3 is master and mac address persistency is infinite

aleabrahao
Meraki Community All-Star
Meraki Community All-Star

@grapevine Palo Alto recommends controlling failover behavior through Link Monitoring and for aggregate interfaces "All" is often required so that failover occurs only when all monitored links are lost.

Refer the documentation.

How to Control Failover on Active/Passive HA for an Aggregate I... - Knowledge Base - Palo Alto Networks

I am not a Cisco employee. My suggestions are based on documentation of Meraki best practices and day-to-day experience.

Please, if this post was useful, leave your kudos and mark it as solved.