02-25-2012 04:47 PM - edited 03-07-2019 05:11 AM
We have 2 6513's that are linked via 2 10 gig interfaces, using an LACP channel.
I received an alert this aft stating that the far 6513 was unreachable and the port channel int PO3 had gone down, the 2 10 gig interfaces had also gone down on either side. 5 mins later PO3 had resestablished itself and has been fine since. I am trying to determine what happened.
Here is a port channel summary from both switches.
Switch 1
Port-channel: Po2 (Primary Aggregator)
------------
Age of the Port-channel = 23d:05h:04m:21s
Logical slot/port = 14/1 Number of ports = 4
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Fast-switchover = disabled
Load share deferral = disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------------+------------------+-----------
0 11 Gi3/1 Active 2
1 22 Gi3/9 Active 2
2 44 Gi3/17 Active 2
3 88 Gi3/25 Active 2
Time since last port bundled: 17d:22h:23m:23s Gi3/25
Time since last port Un-bundled: 21d:09h:18m:43s Gi3/1
Group: 3
----------
Group state = L2
Ports: 2 Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol: LACP
Minimum Links: 0
Ports in the group:
-------------------
Port: Te10/2
------------
Port state = Up Mstr In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 0 Load = 0x55 Protocol = LACP
Mode = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te10/2 SA bndl 32768 0x3 0x3 0xA03 0x3D
Partner's information:
Partner Partner LACP Partner Partner Partner Partner Partner
Port Flags State Port Priority Admin Key Oper Key Port Number Port State
Te10/2 SA bndl 32768 0x0 0x2 0x903 0x3D
Age of the port in the current state: 22d:23h:07m:27s
Port: Te10/3
------------
Port state = Up Mstr In-Bndl
Channel group = 3 Mode = Active Gcchange = -
Port-channel = Po3 GC = - Pseudo port-channel = Po3
Port index = 1 Load = 0xAA Protocol = LACP
Mode = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te10/3 SA bndl 32768 0x3 0x3 0xA04 0x3D
Partner's information:
Partner Partner LACP Partner Partner Partner Partner Partner
Port Flags State Port Priority Admin Key Oper Key Port Number Port State
Te10/3 SA bndl 32768 0x0 0x2 0x904 0x3D
Age of the port in the current state: 22d:23h:07m:28s
Port-channels in the group:
----------------------
Port-channel: Po3 (Primary Aggregator)
------------
Age of the Port-channel = 23d:05h:04m:22s
Logical slot/port = 14/2 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Fast-switchover = disabled
Load share deferral = disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------------+------------------+-----------
0 55 Te10/2 Active 4
1 AA Te10/3 Active 4
Time since last port bundled: 0d:05h:58m:32s Te10/3
Time since last port Un-bundled: 0d:06h:00m:05s Te10/3
Switch 2
Group: 2
----------
Group state = L2
Ports: 2 Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol: LACP
Minimum Links: 0
Ports in the group:
-------------------
Port: Te9/2
------------
Port state = Up Mstr In-Bndl
Channel group = 2 Mode = Active Gcchange = -
Port-channel = Po2 GC = - Pseudo port-channel = Po2
Port index = 0 Load = 0x55 Protocol = LACP
Mode = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te9/2 SA bndl 32768 0x2 0x2 0x903 0x3D
Partner's information:
Partner Partner LACP Partner Partner Partner Partner Partner
Port Flags State Port Priority Admin Key Oper Key Port Number Port State
Te9/2 SA bndl 32768 0x0 0x3 0xA03 0x3D
Age of the port in the current state: 15d:05h:26m:56s
Port: Te9/3
------------
Port state = Up Mstr In-Bndl
Channel group = 2 Mode = Active Gcchange = -
Port-channel = Po2 GC = - Pseudo port-channel = Po2
Port index = 1 Load = 0xAA Protocol = LACP
Mode = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te9/3 SA bndl 32768 0x2 0x2 0x904 0x3D
Partner's information:
Partner Partner LACP Partner Partner Partner Partner Partner
Port Flags State Port Priority Admin Key Oper Key Port Number Port State
Te9/3 SA bndl 32768 0x0 0x3 0xA04 0x3D
Age of the port in the current state: 0d:06h:00m:08s
Port-channels in the group:
----------------------
Port-channel: Po2 (Primary Aggregator)
------------
Age of the Port-channel = 15d:11h:25m:37s
Logical slot/port = 14/1 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Fast-switchover = disabled
Load share deferral = disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------------+------------------+-----------
0 55 Te9/2 Active 4
1 AA Te9/3 Active 4
Time since last port bundled: 0d:06h:00m:10s Te9/3
Time since last port Un-bundled: 0d:06h:01m:44s Te9/3
02-25-2012 07:19 PM
Greetings.
Did either 6500's switch logs show anything? 5 min down unexpectedly, then re-establishing without any intervention sounds like a possible err-disable with recovery...
Matt
02-27-2012 05:05 AM
I turned up the logging last night and it happened again this morn Looks like Mod 9 card is resetting, any ideas this card has decided to start resetting itself ?
2w2d: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 9 to recover from error: Linecard received system exception. Errcode =
2709000001
2w2d: %OIR-SP-3-PWRCYCLE: Card in module 9, is being power-cycled 'Off (Module Reset due to exception or user request)'
2w2d: %C6KPWR-SP-4-DISABLED: power to module in slot 9 set Off (Module Reset due to exception or user request)
*Mar 1 00:00:02.331: DaughterBoard (Centralized Forwarding Card)
Firmware compiled 15-Nov-11 14:23 by integ Build [100]
Feb 27 10:43:53.927: %SYS-CFC9-5-RESTART: System restarted --
Cisco IOS Software, c6lc2 Software (c6lc2-SP-M), Version 12.2(33)SXJ2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 14-Dec-11 19:50 by prod_rel_team
Feb 27 10:43:53.931: CFC9: Currently running ROMMON from S (Gold) region
2w2d: %DIAG-SP-6-RUN_MINIMUM: Module 9: Running Minimal Diagnostics...
2w2d: %DIAG-SP-6-DIAG_OK: Module 9: Passed Online Diagnostics
2w2d: %OIR-SP-6-INSCARD: Card inserted in slot 9, interfaces are now online
02-27-2012 06:44 AM
You can try reseating the module that is giving problems, or you can RMA it and get a replacement. Probably the easiest solution.
If you do get a replacement and it still resets itself then it could be a chassis issue but it is more likely to be the card.
Jon
02-27-2012 07:13 AM
2w2d: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 9 to recover from error: Linecard received system exception. Errcode =
2709000001
Can you connect console to the line card and see if it is crashing?. Crash file generated can be helpful to debug. Meanwhile, getting replacement from TAC is best option.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide