Ping drop

[grapevine]

We have a stack of 3 cisco switches in stack connected to Palo Alto in HA in LACP etherchannel
Firewall 1 has links to Gi1/0/23 and Gi3/0/11
Firewall 2 has links to Gi2/0/23 and Gi3/0/13
We have lacp rate as slow and configured lacp fast-switchover on interface portchannels
We did a failover testing and when switch 3 was powered off firewall is bringing all portchannel-links down on the active firewall and failing over to passive
this is causing a lot of ping drops
When switch 2 and switch 1 are powered off when active, we see only one ping drop
We shut the ports Gi3/0/11 and Gi3/0/13 one by one and did the failover testing still the same results
We configured lacp rate fast and spanning tree portfast trunk on the port channel still the same issue
Please help us fix this

[mloraditch]

Do the Palo Alto logs show anything different between when SW3 is removed vs SW1 or SW2? Based on your topology, removing SW3 kills a link on both PAs at once. I suspect something is renegotiating in that instance that doesn't happen when you just pull SW1 or SW2.

[Joseph W. Doherty]

Is SW3 the active master?

Is your stack also configured not to change the stack's MAC if there is a stack master failure?

BTW, it's often helpful if you identify the specific switch models and the IOS version being used.

[Giuseppe Larosa]

Hello @grapevine ,

first of all check who is the master in the stack with

show switch

as suggested you should implement MAC address persistency so that also in case of change of master there are no impacts also in STP.

Hope to help

Giuseppe

[Joseph W. Doherty]

@grapevine as you haven't answered my questions, cannot with high assurance answer your questions.  However, what I suspect (likely so does @Giuseppe Larosa ) there can be a noticeable difference in operational impact between a non stack member failing vs. the stack master failing.  The latter, i.e. a stack master failure, can be more impactful, as it's the operational brains of the stack.

One issue, on many stacks, if the stack master fails, by default, the stack uses a new MAC for stack operation.  However, there's often a configuration option to retain the original stack MAC even when the stack master fails.  This often mitigate the impact of a stack master failure.

BTW, if the stack is being used for L3 operation, there are are often (?) other configuration options to minimize the impact of a stack master failure for those too.

Again, though, as you didn't answer my questions, cannot say with much certainty the cause of the behavior you note, but I can say, from experience, loss of a stack master can be more operationally impactful, especially if you've not configured the options to mitigate its impact. 

[grapevine]

I also see both links in portchannel on firewall going down and causing firewall failover as well

[Joseph W. Doherty]

@grapevine , again, it can be very helpful when you provide additional base information, such as the actual physical devices being used and the IOS version being used.  (Sometimes it helps others find a documented bug that covers the issue you're encountering.). Further, answering any specific requests for specific information.

So far, you've confirmed the ping loss, and drops of Etherchannel links, on FW, when stack master is powered off (?).

From what you describe, the impact of loss of stack master seems more impactful than I would have expected.  Assuming a bug is not involved, I would suspect, possibly, the configuration isn't optimal.  I have in mind SSO (I believe this is mostly/fully "automatic" for a stack), NSF (often needed for L3 protocols, like routing, to keep neighbor relationships active during fail over recovery - also may require neighbors to be NSF aware [FW?]) and possibly LACP parameters too.

From your description of ping drops, you didn't describe whether ping loss was for stack as source or destination, or all ping traffic was transit, or different results for transit traffic vs. local src/dst.

The big issue appears to be Etherchannel links seen going down on FW.  I've no experience with a Palo Alto FW, and possibly, it and a Cisco physical stack might not ever play nicely together if certain physical stack member failures.  Possibly, the FW needs configuration changes.

Assuming the PA FW and Cisco physical stack cannot cooperate, one or both, unfortunately, might need to be replaced.  On the Cisco side, possibly a Stackwise Virtual might work with the FW.

Have you contacted either/both Palo Alto and/or Cisco TAC?  What you're trying to accomplish isn't oddball and both vendors should have more experience with this kind of requirement.

[grapevine]

I have opened a case with Palo alto TAC and waiting to hear from them.

Debug logs from the switch

Jul 14 20:30:50.085: %PLATFORM-6-HASTATUS: RP switchover, received chassis event to become active
Jul 14 20:30:50.155: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_NOT_PRESENT)
Jul 14 20:30:50.155: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_DOWN)
Jul 14 20:30:50.156: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_REDUNDANCY_STATE_CHANGE)
Jul 14 20:30:50.194: %PLATFORM-6-HASTATUS: RP switchover, sent message became active. IOS is ready to switch to primary after chassis confirmation
Jul 14 20:30:49.896: %STACKMGR-6-STACK_LINK_CHANGE: Switch 4 R0/0: stack_mgr: Stack port 2 on Switch 4 is down
Jul 14 20:30:49.897: %STACKMGR-4-SWITCH_REMOVED: Switch 4 R0/0: stack_mgr: Switch 3 has been removed from the stack.
Jul 14 20:30:50.265: %PLATFORM-6-HASTATUS: RP switchover, received chassis event became active
Jul 14 20:30:49.938: %STACKMGR-4-SWITCH_REMOVED: Switch 1 R0/0: stack_mgr: Switch 3 has been removed from the stack.
Jul 14 20:30:49.769: %STACKMGR-4-SWITCH_REMOVED: Switch 2 R0/0: stack_mgr: Switch 3 has been removed from the stack.
Jul 14 20:30:50.454: %PLATFORM-6-HASTATUS_DETAIL: RP switchover, received chassis event became active. Switch to primary (count 1)
Jul 14 20:30:50.513: %HA-6-SWITCHOVER: Route Processor switched from standby to being active
Jul 14 20:30:51.388: %NDBMAN-5-ACTIVE: Switch 1 R0/0: ndbmand: All data providers active.
Jul 14 20:31:43.062: %DMI-5-SYNC_NEEDED: Switch 1 R0/0: dmiauthd: Configuration change requiring running configuration sync detected - 'Initial sync'. The running configuration will be synchronized to the NETCONF running data store.
Jul 14 20:31:44.164: %DMI-5-SYNC_START: Switch 1 R0/0: dmiauthd: Synchronization of the running configuration to the NETCONF running data store has started.
Jul 14 20:32:07.605: %ETC-5-L3DONTBNDL2: Gi1/0/26 suspended: LACP currently not enabled on the remote port.
Jul 14 20:32:25.867: %ETC-5-L3DONTBNDL2: Gi2/0/26 suspended: LACP currently not enabled on the remote port.
Jul 14 20:33:33.167: %LINK-3-UPDOWN: Interface Lsmpi10/3, changed state to up
Jul 14 20:33:33.168: %LINK-3-UPDOWN: Interface EOBC10/1, changed state to up
Jul 14 20:33:33.169: %LINK-3-UPDOWN: Interface LIIN10/2, changed state to up
Jul 14 20:33:33.171: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/1, changed state to down
Jul 14 20:33:33.172: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/3, changed state to down
Jul 14 20:33:33.172: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/4, changed state to down
Jul 14 20:33:33.172: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/5, changed state to down
Jul 14 20:33:33.172: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/6, changed state to down
Jul 14 20:33:33.172: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/8, changed state to down
Jul 14 20:33:33.172: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/11, changed state to down
Jul 14 20:33:33.173: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/12, changed state to down
Jul 14 20:33:33.173: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/13, changed state to down
Jul 14 20:33:33.173: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/14, changed state to down
Jul 14 20:33:33.173: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/17, changed state to down
Jul 14 20:33:33.173: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/18, changed state to down
Jul 14 20:33:33.173: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to down
Jul 14 20:33:33.173: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/22, changed state to down
Jul 14 20:33:33.174: %LINK-3-UPDOWN: Interface AppGigabitEthernet3/0/1, changed state to down
Jul 14 20:33:33.174: %LINK-3-UPDOWN: Interface Port-channel1, changed state to down
Jul 14 20:33:33.176: %LINK-3-UPDOWN: Interface Port-channel2, changed state to down
Jul 14 20:33:33.430: %LINK-3-UPDOWN: Interface Null0, changed state to up
Jul 14 20:33:33.432: %LINK-3-UPDOWN: Interface Vlan1, changed state to up
Jul 14 20:33:33.449: %LINK-3-UPDOWN: Interface Vlan10, changed state to up
Jul 14 20:33:33.453: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Jul 14 20:33:33.455: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Jul 14 20:33:33.457: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up
Jul 14 20:33:33.459: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to up
Jul 14 20:33:33.461: %LINK-3-UPDOWN: Interface AppGigabitEthernet1/0/1, changed state to up
Jul 14 20:33:33.462: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to up
Jul 14 20:33:33.464: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to up
Jul 14 20:33:33.466: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to up
Jul 14 20:33:33.468: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/5, changed state to up
Jul 14 20:33:33.470: %LINK-3-UPDOWN: Interface AppGigabitEthernet2/0/1, changed state to up
Jul 14 20:33:33.472: %LINK-3-UPDOWN: Interface AppGigabitEthernet4/0/1, changed state to up
Jul 14 20:33:34.165: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi10/3, changed state to up
Jul 14 20:33:34.166: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC10/1, changed state to up
Jul 14 20:33:34.429: %LINEPROTO-5-UPDOWN: Line protocol on Interface Null0, changed state to up
Jul 14 20:33:34.429: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
Jul 14 20:33:34.441: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan10, changed state to up
Jul 14 20:33:34.455: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Jul 14 20:33:34.456: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up
Jul 14 20:33:34.456: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
Jul 14 20:33:34.457: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to up
Jul 14 20:33:34.462: %LINEPROTO-5-UPDOWN: Line protocol on Interface AppGigabitEthernet1/0/1, changed state to up
Jul 14 20:33:34.463: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/1, changed state to up
Jul 14 20:33:34.464: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/2, changed state to up
Jul 14 20:33:34.465: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/3, changed state to up
Jul 14 20:33:34.465: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/5, changed state to up
Jul 14 20:33:34.470: %LINEPROTO-5-UPDOWN: Line protocol on Interface AppGigabitEthernet2/0/1, changed state to up
Jul 14 20:33:34.472: %LINEPROTO-5-UPDOWN: Line protocol on Interface AppGigabitEthernet4/0/1, changed state to up
Applying config on Switch 2...[DONE]
Jul 14 20:33:38.296: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 3 has been added to the stack.
Jul 14 20:33:38.259: %STACKMGR-4-SWITCH_ADDED: Switch 4 R0/0: stack_mgr: Switch 3 has been added to the stack.
Jul 14 20:33:38.210: %STACKMGR-4-SWITCH_ADDED: Switch 2 R0/0: stack_mgr: Switch 3 has been added to the stack.
Jul 14 20:33:39.369: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/26, changed state to up
Jul 14 20:33:43.537: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/26, changed state to up
Jul 14 20:33:44.530: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
Jul 14 20:33:44.717: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/26, changed state to up
Applying config on Switch 4
Jul 14 20:33:45.529: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up...[DONE]
Jul 14 20:33:45.898: %POWER_SUPPLIES-3-PWR_FAIL: Power supply 1 is not functioning for switch 3
Jul 14 20:33:45.898: %POWER_SUPPLIES-3-PWR_FAIL: Power supply 2 is not functioning for switch 3
Jul 14 20:33:46.437: %DMI-5-SYNC_NEEDED: Switch 1 R0/0: dmiauthd: Configuration change requiring running configuration sync detected - 'sync upon syslog event'. The running configuration will be synchronized to the NETCONF running data store.
Jul 14 20:33:47.609: %DMI-5-SYNC_START: Switch 1 R0/0: dmiauthd: Synchronization of the running configuration to the NETCONF running data store has started.
Jul 14 20:33:49.617: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/26, changed state to up
Jul 14 20:33:49.897: %POWER_SUPPLIES-5-PWR_OK: Power supply 1 is functioning for switch 3
Jul 14 20:33:49.898: %POWER_SUPPLIES-5-PWR_OK: Power supply 2 is functioning for switch 3
Jul 14 20:33:50.617: %LINK-3-UPDOWN: Interface Port-channel2, changed state to up
Jul 14 20:33:51.617: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel2, changed state to up
Applying config on Switch 3...[DONE]
Jul 14 20:33:55.319: %DMI-5-SYNC_NEEDED: Switch 1 R0/0: dmiauthd: Configuration change requiring running configuration sync detected - 'sync upon syslog event'. The running configuration will be synchronized to the NETCONF running data store.
Jul 14 20:33:30.799: %STACKMGR-6-STACK_LINK_CHANGE: Switch 3 R0/0: stack_mgr: Stack port 1 on Switch 3 is down
Jul 14 20:33:30.800: %STACKMGR-6-STACK_LINK_CHANGE: Switch 3 R0/0: stack_mgr: Stack port 2 on Switch 3 is down
Jul 14 20:33:32.286: %STACKMGR-6-STACK_LINK_CHANGE: Switch 3 R0/0: stack_mgr: Stack port 1 on Switch 3 is up
Jul 14 20:33:32.288: %STACKMGR-6-STACK_LINK_CHANGE: Switch 3 R0/0: stack_mgr: Stack port 2 on Switch 3 is up
Jul 14 20:33:32.338: %STACKMGR-4-SWITCH_ADDED: Switch 3 R0/0: stack_mgr: Switch 3 has been added to the stack.
Jul 14 20:33:37.287: %STACKMGR-4-SWITCH_ADDED: Switch 3 R0/0: stack_mgr: Switch 3 has been added to the stack.
Jul 14 20:34:11.837: %PSD_MOD-5-DMI_NOTIFY_RESTCONF_START: Switch 2 R0/0: psd: PSD/DMI: restconf server has been notified to start
Jul 14 20:34:12.542: %SPANTREE-5-TOPOTRAP: Topology Change Trap for vlan 10
Jul 14 20:34:16.589: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
Jul 14 20:34:17.309: %LINK-3-UPDOWN: Interface AppGigabitEthernet3/0/1, changed state to up
Jul 14 20:34:17.609: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Jul 14 20:34:17.781: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/1, changed state to up
Jul 14 20:34:17.873: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/17, changed state to up
Jul 14 20:34:17.923: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/3, changed state to up
Jul 14 20:34:17.925: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/18, changed state to up
Jul 14 20:34:18.022: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to up
Jul 14 20:34:18.114: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/4, changed state to up
Jul 14 20:34:18.209: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/13, changed state to up
Jul 14 20:34:18.258: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/11, changed state to up
Jul 14 20:34:18.305: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/6, changed state to up
Jul 14 20:34:18.312: %LINEPROTO-5-UPDOWN: Line protocol on Interface AppGigabitEthernet3/0/1, changed state to up
Jul 14 20:34:18.367: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/14, changed state to up
Jul 14 20:34:18.410: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/12, changed state to up
Jul 14 20:34:18.462: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/5, changed state to up
Jul 14 20:34:18.616: %SPANTREE-5-TOPOTRAP: Topology Change Trap for vlan 10
Jul 14 20:34:18.782: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/1, changed state to up
Jul 14 20:34:18.873: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/17, changed state to up
Jul 14 20:34:18.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/18, changed state to up
Jul 14 20:34:19.022: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to up
Jul 14 20:34:19.304: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/6, changed state to up
Jul 14 20:34:19.366: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/14, changed state to up
Jul 14 20:34:19.409: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/12, changed state to up
Jul 14 20:34:19.462: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/5, changed state to up
Jul 14 20:34:19.926: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/3, changed state to up
Jul 14 20:34:20.118: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/4, changed state to up
Jul 14 20:34:22.401: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/13, changed state to up
Jul 14 20:34:22.769: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/11, changed state to up
Jul 14 20:34:28.322: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
Jul 14 20:35:03.968: %DMI-5-SYNC_COMPLETE: Switch 1 R0/0: dmiauthd: The running configuration has been synchronized to the NETCONF running data store.
Jul 14 20:35:04.782: %DMI-5-SYNC_START: Switch 1 R0/0: dmiauthd: Synchronization of the running configuration to the NETCONF running data store has started.
Jul 14 20:35:07.270: %SSH-5-SSH2_SESSION: SSH2 Session request from 10.253.4.214 (tty = 0) using crypto cipher 'aes256-ctr', hmac '[email protected]' Succeeded
Jul 14 20:35:37.711: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: admin] [Source: 10.253.4.214] [localport: 22] at 13:35:37 PDT Tue Jul 14 2026
Jul 14 20:35:37.711: %SSH-5-SSH2_USERAUTH: User 'admin' authentication for SSH2 Session from 10.253.4.214 (tty = 0) using crypto cipher 'aes256-ctr', hmac '[email protected]' Succeeded
Jul 14 20:35:53.966: %SYS-6-LOGOUT: User admin has exited tty session 1(10.253.4.214)
Jul 14 20:35:53.967: %SSH-5-SSH2_CLOSE: SSH2 Session from 10.253.4.214 (tty = 0) for user '' using crypto cipher 'aes256-ctr', hmac '[email protected]' closed
Jul 14 20:36:17.922: %DMI-5-SYNC_COMPLETE: Switch 1 R0/0: dmiauthd: The running configuration has been synchronized to the NETCONF running data store.

[grapevine]

We run Cisco IOS-XE 17.12.05

[grapevine]

sw3 is master and mac address persistency is infinite

[aleabrahao]

@grapevine Palo Alto recommends controlling failover behavior through Link Monitoring and for aggregate interfaces "All" is often required so that failover occurs only when all monitored links are lost.

Refer the documentation.

How to Control Failover on Active/Passive HA for an Aggregate I... - Knowledge Base - Palo Alto Networks