VSL link failure

[leofalconi]

Hello people,

I had a problem with WS-C4506-E last Saturday.

There are these logs:

%VSLP-3-VSLP_LMP_FAIL_REASON: Te1/1/1: Link down
Feb 25 21:24:47.515: %VSLP-2-VSL_DOWN:   All VSL links went down while switch is in ACTIVE role

Feb 25 21:24:47.515: %EC-5-UNBUNDLE: Interface TenGigabitEthernet2/1/1 left the port-channel Port-channel2
Feb 25 21:24:47.676: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been lost
Feb 25 21:24:47.690: %EC-5-UNBUNDLE: Interface Te2/2/1 left the port-channel Po11
Feb 25 21:24:47.691: %EC-5-UNBUNDLE: Interface Te2/2/2 left the port-channel Po12
Feb 25 21:24:47.693: %EC-5-UNBUNDLE: Interface Te2/2/3 left the port-channel Po13
Feb 25 21:24:47.694: %EC-5-UNBUNDLE: Interface Te2/2/4 left the port-channel Po14
Feb 25 21:24:47.695: %EC-5-UNBUNDLE: Interface Te2/2/5 left the port-channel Po10
Feb 25 21:24:47.697: %EC-5-UNBUNDLE: Interface Te2/2/12 left the port-channel Po62
Feb 25 21:24:47.699: %EC-5-UNBUNDLE: Interface Te2/3/1 left the port-channel Po15
Feb 25 21:24:47.700: %EC-5-UNBUNDLE: Interface Te2/3/2 left the port-channel Po16
Feb 25 21:24:47.701: %EC-5-UNBUNDLE: Interface Te2/3/3 left the port-channel Po17
Feb 25 21:24:47.703: %EC-5-UNBUNDLE: Interface Te2/3/4 left the port-channel Po18
Feb 25 21:24:47.704: %EC-5-UNBUNDLE: Interface Te2/3/5 left the port-channel Po19
Feb 25 21:24:47.706: %EC-5-UNBUNDLE: Interface Te2/3/11 left the port-channel Po61
Feb 25 21:24:47.707: %EC-5-UNBUNDLE: Interface Te2/3/12 left the port-channel Po50
Feb 25 21:24:47.730: %EC-5-UNBUNDLE: Interface Gi2/6/22 left the port-channel Po53
Feb 25 21:24:47.732: %EC-5-UNBUNDLE: Interface Gi2/6/23 left the port-channel Po53
Feb 25 21:24:47.734: %EC-5-UNBUNDLE: Interface Gi2/6/25 left the port-channel Po54
Feb 25 21:24:47.735: %EC-5-UNBUNDLE: Interface Gi2/6/26 left the port-channel Po54
Feb 25 21:24:47.738: %EC-5-UNBUNDLE: Interface Gi2/6/41 left the port-channel Po102
Feb 25 21:24:47.740: %EC-5-UNBUNDLE: Interface Gi2/6/42 left the port-channel Po101
Feb 25 21:24:47.743: %FASTHELLO-2-FH_DOWN:  Fast-Hello interface Gi2/6/48 lost dual-active detection capability

Feb 25 21:24:47.775: %C4K_REDUNDANCY-3-SIMPLEX_MODE: The peer Supervisor has been lost
Feb 25 21:24:48.026: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session DOWN on slot 6 port 48.
Feb 25 21:24:54.963: %FASTHELLO-2-FH_DOWN:  Fast-Hello interface Gi1/6/48 lost dual-active detection capability

Feb 25 21:25:01.330: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 10.119.0.16 (Vlan7) is down: holding time expired
Feb 25 21:27:41.290: %SFF8472-5-THRESHOLD_VIOLATION: Te1/1/1: Rx power low alarm; Operating value: -26.9 dBm, Threshold value: -13.9 dBm.
Feb 25 21:27:41.291: %SFF8472-5-THRESHOLD_VIOLATION: Te1/1/2: Rx power low alarm; Operating value: -40.0 dBm, Threshold value: -13.9 dBm.
Feb 25 21:29:58.226: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session UP on slot 1 port 1.
Feb 25 21:29:58.429: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session UP on slot 1 port 2.
Feb 25 21:30:14.232: %VSLP-5-VSL_UP:  Ready for control traffic

Feb 25 21:30:18.226: %VSLP-5-RRP_ROLE_RESOLVED: Role resolved as ACTIVE  by VSLP
Feb 25 21:30:18.226: %EC-5-BUNDLE: Interface TenGigabitEthernet1/1/1 joined port-channel Port-channel1
Feb 25 21:30:18.239: %EC-5-BUNDLE: Interface TenGigabitEthernet1/1/2 joined port-channel Port-channel1
Feb 25 21:30:18.803: %C4K_REDUNDANCY-6-DUPLEX_MODE: The peer Supervisor has been detected
Feb 25 21:30:57.648: %C4K_REDUNDANCY-6-MODE: ACTIVE supervisor initializing for sso mode
Feb 25 21:30:57.814: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been established
Feb 25 21:31:08.406: %C4K_REDUNDANCY-5-CONFIGSYNC: The bootvar has been successfully synchronized to the standby supervisor
Feb 25 21:31:08.407: %C4K_REDUNDANCY-5-CONFIGSYNC: The config-reg has been successfully synchronized to the standby supervisor
Feb 25 21:31:08.408: %C4K_REDUNDANCY-5-CONFIGSYNC: The startup-config has been successfully synchronized to the standby supervisor
Feb 25 21:31:09.193: %C4K_REDUNDANCY-5-CONFIGSYNC: The private-config has been successfully synchronized to the standby supervisor
Feb 25 21:31:10.455: %C4K_REDUNDANCY-5-CONFIGSYNC_RATELIMIT: The vlan database has been successfully synchronized to the standby supervisor
Feb 25 21:31:50.522: %EC-5-BUNDLE: Interface TenGigabitEthernet2/1/1 joined port-channel Port-channel2
Feb 25 21:31:50.554: %EC-5-BUNDLE: Interface TenGigabitEthernet2/1/2 joined port-channel Port-channel2
Feb 25 21:31:53.379: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
Feb 25 21:31:52.901: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 1 (WS-X45-SUP7-E S/N: CAT1910L2Y9 Hw: 3.0) is online
Feb 25 21:31:52.901: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 2 (WS-X4712-SFP+E S/N: CAT1910L0G1 Hw: 2.0) is online
Feb 25 21:31:52.901: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 3 (WS-X4712-SFP+E S/N: CAT1910L0MH Hw: 2.0) is online
Feb 25 21:31:52.901: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 6 (WS-X4748-RJ45-E S/N: CAT1911L195 Hw: 1.1) is online
Feb 25 21:31:52.901: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 11 (WS-X45-SUP7-E S/N: CAT1910L31H Hw: 3.0) is online
Feb 25 21:31:54.374: %RF-5-RF_TERMINAL_STATE: 1 ha_mgr:  Terminal state reached for (SSO)
Feb 25 21:31:52.901: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 12 (WS-X4712-SFP+E S/N: CAT1910L0G0 Hw: 2.0) is online
Feb 25 21:31:52.902: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 13 (WS-X4712-SFP+E S/N: CAT1910L0L3 Hw: 2.0) is online
Feb 25 21:31:52.902: %C4K_IOSMODPORTMAN-6-MODULEONLINE: STANDBY:Module 16 (WS-X4748-RJ45-E S/N: CAT1911L168 Hw: 1.1) is online
Feb 25 21:31:56.524: %FASTHELLO-5-FH_UP:  Fast-Hello interface Gi1/6/48 status changed to link up

Feb 25 21:31:56.633: %FASTHELLO-5-FH_UP:  Fast-Hello interface Gi2/6/48 status changed to link up

Feb 25 21:31:56.683: %FASTHELLO-5-FH_DUAL_ACTIVE_CAP:  Fast-Hello interface Gi1/6/48 became dual-active detection capable

Feb 25 21:31:55.679: %FASTHELLO-5-FH_UP: STANDBY: Fast-Hello interface Gi1/6/48 status changed to link up

Feb 25 21:31:55.767: %FASTHELLO-5-FH_UP: STANDBY: Fast-Hello interface Gi2/6/48 status changed to link up

Feb 25 21:31:55.868: %FASTHELLO-5-FH_DUAL_ACTIVE_CAP: STANDBY: Fast-Hello interface Gi2/6/48 became dual-active detection capable
Feb 25 21:32:01.375: %EC-5-BUNDLE: Interface Te2/3/5 joined port-channel Po19
Feb 25 21:32:02.048: %EC-5-BUNDLE: Interface Te2/2/5 joined port-channel Po10
Feb 25 21:32:02.212: %EC-5-BUNDLE: Interface Te2/2/2 joined port-channel Po12
Feb 25 21:32:02.215: %EC-5-BUNDLE: Interface Te2/2/1 joined port-channel Po11
Feb 25 21:32:02.251: %EC-5-BUNDLE: Interface Te2/2/4 joined port-channel Po14
Feb 25 21:32:02.370: %EC-5-BUNDLE: Interface Te2/2/12 joined port-channel Po62
Feb 25 21:32:02.391: %EC-5-BUNDLE: Interface Te2/3/11 joined port-channel Po61
Feb 25 21:32:02.489: %EC-5-BUNDLE: Interface Te2/2/3 joined port-channel Po13
Feb 25 21:32:00.613: %EC-5-BUNDLE: STANDBY:Interface Te2/3/5 joined port-channel Po19
Feb 25 21:32:01.332: %EC-5-BUNDLE: STANDBY:Interface Te2/2/2 joined port-channel Po12
Feb 25 21:32:01.415: %EC-5-BUNDLE: STANDBY:Interface Te2/2/1 joined port-channel Po11
Feb 25 21:32:01.488: %EC-5-BUNDLE: STANDBY:Interface Te2/2/4 joined port-channel Po14
Feb 25 21:32:01.510: %EC-5-BUNDLE: STANDBY:Interface Te2/2/12 joined port-channel Po62
Feb 25 21:32:01.546: %EC-5-BUNDLE: STANDBY:Interface Te2/3/11 joined port-channel Po61
Feb 25 21:32:01.613: %EC-5-BUNDLE: STANDBY:Interface Te2/2/5 joined port-channel Po10
Feb 25 21:32:03.088: %EC-5-BUNDLE: Interface Te2/3/3 joined port-channel Po17
Feb 25 21:32:03.146: %EC-5-BUNDLE: Interface Te2/3/2 joined port-channel Po16
Feb 25 21:32:03.269: %EC-5-BUNDLE: Interface Te2/3/1 joined port-channel Po15
Feb 25 21:32:03.317: %EC-5-BUNDLE: Interface Te2/3/12 joined port-channel Po50
Feb 25 21:32:03.499: %EC-5-BUNDLE: Interface Gi2/6/25 joined port-channel Po54
Feb 25 21:32:03.525: %EC-5-BUNDLE: Interface Te2/3/4 joined port-channel Po18
Feb 25 21:32:03.535: %EC-5-BUNDLE: Interface Gi2/6/42 joined port-channel Po101
Feb 25 21:32:01.723: %EC-5-BUNDLE: STANDBY:Interface Te2/2/3 joined port-channel Po13
Feb 25 21:32:02.350: %EC-5-BUNDLE: STANDBY:Interface Te2/3/3 joined port-channel Po17
Feb 25 21:32:02.459: %EC-5-BUNDLE: STANDBY:Interface Te2/3/1 joined port-channel Po15
Feb 25 21:32:02.555: %EC-5-BUNDLE: STANDBY:Interface Te2/3/12 joined port-channel Po50
Feb 25 21:32:02.629: %EC-5-BUNDLE: STANDBY:Interface Te2/3/2 joined port-channel Po16
Feb 25 21:32:02.647: %EC-5-BUNDLE: STANDBY:Interface Gi2/6/25 joined port-channel Po54
Feb 25 21:32:04.234: %EC-5-BUNDLE: Interface Gi2/6/41 joined port-channel Po102
Feb 25 21:32:02.770: %EC-5-BUNDLE: STANDBY:Interface Te2/3/4 joined port-channel Po18
Feb 25 21:32:03.022: %EC-5-BUNDLE: STANDBY:Interface Gi2/6/42 joined port-channel Po101
Feb 25 21:32:03.518: %EC-5-BUNDLE: STANDBY:Interface Gi2/6/41 joined port-channel Po102
Feb 25 21:32:05.318: %EC-5-BUNDLE: Interface Gi2/6/26 joined port-channel Po54
Feb 25 21:32:05.420: %EC-5-BUNDLE: Interface Gi2/6/22 joined port-channel Po53
Feb 25 21:32:05.514: %EC-5-BUNDLE: Interface Gi2/6/23 joined port-channel Po53
Feb 25 21:32:04.536: %EC-5-BUNDLE: STANDBY:Interface Gi2/6/26 joined port-channel Po54
Feb 25 21:32:04.636: %EC-5-BUNDLE: STANDBY:Interface Gi2/6/22 joined port-channel Po53
Feb 25 21:32:04.748: %EC-5-BUNDLE: STANDBY:Interface Gi2/6/23 joined port-channel Po53
Feb 25 21:32:28.998: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 10.119.0.16 (Vlan7) is up: new adjacency
Feb 26 01:14:42.248: %C4K_REDUNDANCY-5-CALENDAR_RATELIMIT: The calendar has been successfully synchronized to the standby supervisor24 times since last calendar syslog
Feb 26 15:32:07.248: %C4K_REDUNDANCY-5-CALENDAR_RATELIMIT: The calendar has been successfully synchronized to the standby supervisor24 times since last calendar syslog
Feb 27 03:52:15.248: %C4K_REDUNDANCY-5-CALENDAR_RATELIMIT: The calendar has been successfully synchronized to the standby supervisor24 times since last calendar syslog

Can you help me to find the cause of these logs?

I think the backup is failed, is possible?

Thank you a lot

Leonardo

[Reza Sharifi]

Hi,

Which switch is primary now?

Can you post the output of "sh redun" and "sh ver"?

HTH

[leofalconi]

SHOW REDUNDANCY:
Redundant System Information :

------------------------------
       Available system uptime = 1 year, 43 weeks, 2 days, 19 hours, 36 minutes
Switchovers system experienced = 0
              Standby failures = 1
        Last switchover reason = none

                 Hardware Mode = Duplex
    Configured Redundancy Mode = Stateful Switchover
     Operating Redundancy Mode = Stateful Switchover
              Maintenance Mode = Disabled
                Communications = Up

Current Processor Information :
------------------------------
               Active Location = slot 1/1
        Current Software state = ACTIVE
       Uptime in current state = 1 year, 43 weeks, 3 days, 2 hours, 30 minutes
                 Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.06.02.E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 25-Mar-15 04:42 by prod
               BOOT = bootflash:cat4500e-universalk9.SPA.03.06.02.E.152-2.E2.bin,1;
        Configuration register = 0x2102

Peer Processor Information :
------------------------------
              Standby Location = slot 2/1
        Current Software state = STANDBY HOT
       Uptime in current state = 2 days, 12 hours, 16 minutes
                 Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.06.02.E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 25-Mar-15 04:42 by pr
               BOOT = bootflash:cat4500e-universalk9.SPA.03.06.02.E.152-2.E2.bin,1;
        Configuration register = 0x2102

SHOW VERSION:

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.06.02.E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 25-Mar-15 04:42 by prod_rel_team



Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.



ROM: 15.0(1r)SG5
MC-SWC-CED01 uptime is 1 year, 43 weeks, 2 days, 19 hours, 37 minutes
Uptime for this control processor is 1 year, 43 weeks, 2 days, 19 hours, 39 minutes
System returned to ROM by power-on
System restarted at 15:11:56 CEST Fri May 1 2015
System image file is "bootflash:cat4500e-universalk9.SPA.03.06.02.E.152-2.E2.bin"
Jawa Revision 7, Snowtrooper Revision 0x0.0x1C

Last reload reason: power-on



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.


License Information for 'WS-X45-SUP7-E'
    License Level: entservices   Type: Permanent
    Next reboot license Level: entservices

cisco WS-C4506-E (MPC8572) processor (revision 11) with 2097152K bytes of physical memory.
Processor board ID FXS1904Q1AB
MPC8572 CPU at 1.5GHz, Supervisor 7
Last reset from PowerUp
36 Virtual Ethernet interfaces
96 Gigabit Ethernet interfaces
56 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2102

Is possible find the cause of this reload? In the flash there isn't any crashinfo and the show version show me only the last reload reason on the active node.

thank you