We have a VSS cluster with 6509e chassis and a single VS-S720-10G in each chassis. We had a failure in the primary chassis which caused a switchover. Now all of the modules in the primary chassis are down and don't seem able to recover. I power cycled the chassis and the supervisor comes up detects VSS and goes into standby mode, but all the modules and interfaces remain down. Here is info from the logs on the active supervisor.
Nov 18 2015 07:55:08.090 EST: %SYS-5-CONFIG_I: Configured from console by <removed> on vty0 (<removed>)
Nov 18 2015 08:05:28.587 EST: %VSLP-SW2_SP-3-VSLP_LMP_FAIL_REASON: Te2/5/5: Link down
Nov 18 2015 08:05:28.587 EST: %VSLP-SW2_SP-2-VSL_DOWN: Last VSL interface Te2/5/5 went down
Nov 18 2015 08:05:28.595 EST: %VSLP-SW2_SP-2-VSL_DOWN: All VSL links went down while switch is in ACTIVE role
Nov 18 2015 08:05:29.575 EST: %PFREDUN-SW2_SP-6-ACTIVE: Standby processor removed or reloaded, changing to Simplex mode
Nov 18 2015 08:05:30.359 EST: %RF-SW2_SP-5-RF_RELOAD: Peer reload. Reason: Proxy request to reload peer
Nov 18 2015 08:07:49.137 EST: %ISSU-SW2_SP-3-PEER_IMAGE_INCOMPATIBLE: Peer image (s72033_sp-IPBASEK9-M), version (12.2.(33)SXI9) on peer uid (-229638123) is incompatible
Nov 18 2015 08:07:49.145 EST: %ISSU-SW2_SP-3-PEER_IMAGE_INCOMPATIBLE: Peer image (s72033_sp-IPBASEK9-M), version (12.2.(33)SXI9) on peer uid (-229638123) is incompatible
Nov 18 2015 08:08:00.121 EST: %VSLP-SW2_SP-5-RRP_ROLE_RESOLVED: Role resolved as ACTIVE by VSLP
Nov 18 2015 08:08:00.121 EST: %VSL-SW2_SP-5-VSL_CNTRL_LINK: New VSL Control Link Te2/5/5
Nov 18 2015 08:08:02.141 EST: %VSLP-SW2_SP-5-VSL_UP: Ready for control traffic
Nov 18 2015 08:08:33.265 EST: %ISSU-SW2_SP-3-PEER_IMAGE_INCOMPATIBLE: Peer image (s72033_sp-IPBASEK9-M), version (12.2.(33)SXI9) on peer uid (-229638123) is incompatible
Nov 18 2015 08:08:33.273 EST: %ISSU-SW2_SP-3-PEER_IMAGE_INCOMPATIBLE: Peer image (s72033_sp-IPBASEK9-M), version (12.2.(33)SXI9) on peer uid (-229638123) is incompatible
Nov 18 2015 08:09:35.590 EST: %PFREDUN-SW2_SP-4-INCOMPATIBLE: Defaulting to RPR mode (Runtime incompatible)
Nov 18 2015 08:09:36.602 EST: %PFREDUN-SW2_SP-6-ACTIVE: Standby initializing for RPR mode
Nov 18 2015 08:09:38.250 EST: %PFINIT-SW2_SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
Nov 18 2015 08:09:42.978 EST: %RF-SW2_SP-5-RF_TERMINAL_STATE: Terminal state reached for (RPR)
Nov 18 2015 08:14:55.663 EST: %SYS-5-CONFIG_I: Configured from console by niemanndadmin on console
Nov 18 2015 08:15:07.963 EST: %PFINIT-SW2_SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
Nov 18 2015 08:16:13.108 EST: %VSLP-SW2_SP-3-VSLP_LMP_FAIL_REASON: Te2/5/5: Link down
Nov 18 2015 08:16:13.108 EST: %VSLP-SW2_SP-2-VSL_DOWN: Last VSL interface Te2/5/5 went down
Nov 18 2015 08:16:13.112 EST: %VSLP-SW2_SP-2-VSL_DOWN: All VSL links went down while switch is in ACTIVE role
Nov 18 2015 08:16:13.776 EST: %PFREDUN-SW2_SP-6-ACTIVE: Standby processor removed or reloaded, changing to Simplex mode
Nov 18 2015 08:16:14.300 EST: %RF-SW2_SP-5-RF_RELOAD: Peer reload. Reason: Proxy request to reload peer
Nov 18 2015 08:18:22.054 EST: %ISSU-SW2_SP-3-PEER_IMAGE_INCOMPATIBLE: Peer image (s72033_sp-IPBASEK9-M), version (12.2.(33)SXI9) on peer uid (-229638123) is incompatible
Nov 18 2015 08:18:22.062 EST: %ISSU-SW2_SP-3-PEER_IMAGE_INCOMPATIBLE: Peer image (s72033_sp-IPBASEK9-M), version (12.2.(33)SXI9) on peer uid (-229638123) is incompatible
Nov 18 2015 08:18:33.036 EST: %VSLP-SW2_SP-5-RRP_ROLE_RESOLVED: Role resolved as ACTIVE by VSLP
Nov 18 2015 08:18:33.036 EST: %VSL-SW2_SP-5-VSL_CNTRL_LINK: New VSL Control Link Te2/5/5
Nov 18 2015 08:18:35.107 EST: %VSLP-SW2_SP-5-VSL_UP: Ready for control traffic
Nov 18 2015 08:19:06.228 EST: %ISSU-SW2_SP-3-PEER_IMAGE_INCOMPATIBLE: Peer image (s72033_sp-IPBASEK9-M), version (12.2.(33)SXI9) on peer uid (-229638123) is incompatible
Nov 18 2015 08:19:06.240 EST: %ISSU-SW2_SP-3-PEER_IMAGE_INCOMPATIBLE: Peer image (s72033_sp-IPBASEK9-M), version (12.2.(33)SXI9) on peer uid (-229638123) is incompatible
Nov 18 2015 08:20:07.459 EST: %PFREDUN-SW2_SP-4-INCOMPATIBLE: Defaulting to RPR mode (Runtime incompatible)
Nov 18 2015 08:20:08.471 EST: %PFREDUN-SW2_SP-6-ACTIVE: Standby initializing for RPR mode
Nov 18 2015 08:20:10.123 EST: %PFINIT-SW2_SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
Nov 18 2015 08:20:14.795 EST: %RF-SW2_SP-5-RF_TERMINAL_STATE: Terminal state reached for (RPR)
VSS outputs
show switch virtual dual-active fast-hello
Fast-hello dual-active detection enabled: Yes
Fast-hello dual-active interfaces:
Port Local State Peer Port Remote State
---------------------------------------------------
Gi2/5/3 Link dn - -
show switch virtual dual-active summary
Pagp dual-active detection enabled: Yes
Fast-hello dual-active detection enabled: Yes
No interfaces excluded from shutdown in recovery mode
In dual-active recovery mode: No
sh switch virtual link
VSL Status : UP
VSL Uptime : 19 minutes
VSL SCP Ping : Pass
VSL ICC Ping : Pass
VSL Control Link : Te2/5/5
sh switch virtual redundancy
My Switch Id = 2
Peer Switch Id = 1
Last switchover reason = active unit removed
Configured Redundancy Mode = sso
Operating Redundancy Mode = rpr
Switch 2 Slot 5 Processor Information :
-----------------------------------------------
Current Software state = ACTIVE
Uptime in current state = 14 hours, 46 minutes
Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-IPBASEK9-M), Version 12.2(33)SXI9, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Fri 24-Feb-12 21:38 by prod_rel_team
BOOT = sup-bootdisk:s72033-ipbasek9-mz.122-33.SXI9.bin,1;
Configuration register = 0x2102
Fabric State = ACTIVE
Control Plane State = ACTIVE
Switch 1 Slot 5 Processor Information :
-----------------------------------------------
Current Software state = STANDBY COLD (switchover target)
Uptime in current state = 17 minutes
Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-IPBASEK9-M), Version 12.2(33)SXI9, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Fri 24-Feb-12 21:38 by prod_rel_team
BOOT = sup-bootdisk:s72033-ipbasek9-mz.122-33.SXI9.bin,1;
Configuration register = 0x2102
Fabric State = ACTIVE
Control Plane State = STANDBY
