Pagp has been working but suddenly stopped

[morphotek]

I have 2 Cisco 9500 setup in virtual stack and its been working since 2019 with no issues.  This past week it stopped working properly and now I see when I do a show stackwise-virtual dual-active-detection pagp the below.  I'm not sure why only 1 port is now showing at Dual-Active Detect Capable.  I do have a TAC case open but its taking longer than I would like to resolve.

Pagp dual-active detection enabled: Yes
In dual-active recovery mode: No

Channel group 3
Dual-Active Partner Partner Partner
Port Detect Capable Name Port Version
Te1/0/3 Yes None None 1.1
Te1/0/4 No None None N/A
Te2/0/3 No None None N/A
Te2/0/4 No None None N/A

[MHM Cisco World]

Show pagp internal <<- share this 

[morphotek]

MorCISNetCore#sh pagp internal
Flags: S - Device is sending Slow hello. C - Device is in Consistent state.
A - Device is in Auto mode. d - PAgP is down
Timers: H - Hello timer is running. Q - Quit timer is running.
S - Switching timer is running. I - Interface timer is running.

Channel group 3
Hello Partner PAgP Learning Group
Port Flags State Timers Interval Count Priority Method Ifindex
Te1/0/3 U4/S4 H 30s 0 128 Any 10
Te1/0/4 U4/S4 H 30s 0 128 Any 11
Te2/0/3 U4/S4 H 30s 0 128 Any 39
Te2/0/4 U4/S4 H 30s 0 128 Any 40

[M02@rt37]

Hello @morphotek,

The fact that the "Dual-Active Partner" column shows "None" for all ports suggests that no dual-active partner has been detected for those ports.

Do you verify the physical connections between the switches and the ports in question. Ensure that the cables are securely connected and there are no physical issues with the ports or cables.

[morphotek]

The cables are secure, and I have tried to swap out the GBICs, and the cables, also all for of the physical interfaces show up, up though the port-channel shows down and not connected which is odd with the physical ports up

[MHM Cisco World]

show pagp x neighbor

show pagp 1 counters

check counters first if the port send receive the pagp then there is no physical 
if you get different neighbor for each port in show pagp neighbor then I think you have I afraid  split brain. 

[morphotek]

For both sh pagp x neighbor, and sh pagp 1 counters I get no interfaces configured in the channel group.   The 4 interfaces are each configured with channel-group 3 mode desirable  I did remove and re-add the config on all 4 ports and did a reload on the switch.