Interface Flapping between Catalyst 9606R (StackWise Virtual) and Palo

[patrick-chang]

Hi everyone,

I am experiencing a frustrating interface flapping issue between a Cisco Catalyst 9606R switch pair and a Palo Alto PA-3430 firewall, and I would appreciate any insights or advice from the community.

Network Topology & Environment:

• Switch: Cisco Catalyst 9606R configured with StackWise Virtual (SVL).

• Firewall: Palo Alto PA-3430.

• Interconnection: 10G SFP+ modules with fiber cabling.

• Protocols: LACP / Port-channel is configured between the Switch and the Firewall.

The Issue:

The switch-side interface frequently generates %LINK-3-UPDOWN and %LINEPROTO-5-UPDOWN logs. Currently, the frequency has dropped to about 1 to 2 times a day, but in a production environment, even a single flap causes unwanted failovers or brief disruptions.

Troubleshooting Done So Far (Physical Layer Excluded):

We have performed extensive hardware and physical layer troubleshooting on the Cisco side to isolate the issue:

1. Changed Switch Ports: Moved the connections to different ports on the same switch member. (Issue persisted)

2. Tested Across SVL Members: Moved the firewall connection from Member 1 to Member 2. (Issue persisted

3. Firewall Spare Unit Test (The Weird Part): When we temporarily swapped the production PA-3430 with our spare PA-3430 firewall, the interface became perfectly stable and no up/down logs were observed at all.

4. Replaced SFPs & Fiber Cables: We swapped out all the SFP+ transceivers and fiber patch cords on both ends, but it did not help.

5. PA RMA Replacement: We have already engaged Palo Alto Support and processed an RMA replacement for the production firewall. However, the issue persists even with the new RMA device.

6. Firewall Spare Unit Test (The Weird Part): Here is the strangest twist—when we temporarily brought our own spare PA-3430 firewall online using the exact same ports, optics, and cables, the interface became perfectly stable and no up/down logs were observed at all.

Questions:

Since the issue disappears when the spare firewall is used, it highly points to the production firewall hardware/transceiver or a specific software bug. However, the PA TAC is still pushing us to perform a loopback test on the Cisco C9606R SVL production ports to completely rule out the switch.

1. Has anyone encountered any known interoperability issues, optics compatibility bugs, or LACP timer/negotiation anomalies between C9606 (IOS-XE) and Palo Alto (PAN-OS) on 10G interfaces?

2. Is there any specific hidden command or behavior regarding FEC/Auto-negotiation on C9606R 10G ports that could cause intermittent link flaps with PA firewalls?

3. Are there any specific debug logs  I should collect on the Cisco side during the next flap to prove the switch is just reacting to the remote link going down?

Any suggestions or historical cases would be greatly appreciated. Thank you in advance!