cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2101
Views
0
Helpful
4
Replies

Lost port channel between 6513's - Need help finding out why

dclee
Level 1
Level 1

We have 2 6513's that are linked via 2 10 gig interfaces, using an LACP channel.

I received an alert this aft stating that the far 6513 was unreachable and the port channel int PO3 had gone down, the 2 10 gig interfaces had also gone down on either side. 5 mins later PO3 had resestablished itself and has been fine since. I am trying to determine what happened.

Here is a port channel summary from both switches.

Switch 1

Port-channel: Po2    (Primary Aggregator)

------------

Age of the Port-channel   = 23d:05h:04m:21s
Logical slot/port   = 14/1          Number of ports = 4
HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP
Fast-switchover     = disabled
Load share deferral = disabled

Ports in the Port-channel:

Index   Load      Port          EC state       No of bits
------+------+------------+------------------+-----------
0      11          Gi3/1             Active   2
1      22          Gi3/9             Active   2
2      44         Gi3/17             Active   2
3      88         Gi3/25             Active   2

Time since last port bundled:    17d:22h:23m:23s    Gi3/25
Time since last port Un-bundled: 21d:09h:18m:43s    Gi3/1

Group: 3
----------
Group state = L2
Ports: 2   Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol:   LACP
Minimum Links: 0
                Ports in the group:
                -------------------
Port: Te10/2
------------

Port state    = Up Mstr In-Bndl
Channel group = 3           Mode = Active      Gcchange = -
Port-channel  = Po3         GC   =   -         Pseudo port-channel = Po3
Port index    = 0           Load = 0x55        Protocol =   LACP
Mode = LACP

Flags:  S - Device is sending Slow LACPDUs   F - Device is sending fast LACPDUs.
        A - Device is in active mode.        P - Device is in passive mode.

Local information:
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Te10/2    SA      bndl       32768         0x3       0x3     0xA03       0x3D

Partner's information:

          Partner Partner   LACP Partner  Partner   Partner  Partner     Partner
Port      Flags   State     Port Priority Admin Key Oper Key Port Number Port State
Te10/2    SA      bndl       32768         0x0       0x2      0x903       0x3D

Age of the port in the current state: 22d:23h:07m:27s

Port: Te10/3
------------

Port state    = Up Mstr In-Bndl
Channel group = 3           Mode = Active      Gcchange = -
Port-channel  = Po3         GC   =   -         Pseudo port-channel = Po3
Port index    = 1           Load = 0xAA        Protocol =   LACP
Mode = LACP

Flags:  S - Device is sending Slow LACPDUs   F - Device is sending fast LACPDUs.
        A - Device is in active mode.        P - Device is in passive mode.

Local information:
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Te10/3    SA      bndl       32768         0x3       0x3     0xA04       0x3D

Partner's information:

          Partner Partner   LACP Partner  Partner   Partner  Partner     Partner
Port      Flags   State     Port Priority Admin Key Oper Key Port Number Port State
Te10/3    SA      bndl       32768         0x0       0x2      0x904       0x3D

Age of the port in the current state: 22d:23h:07m:28s

                Port-channels in the group:
                ----------------------

Port-channel: Po3    (Primary Aggregator)

------------

Age of the Port-channel   = 23d:05h:04m:22s
Logical slot/port   = 14/2          Number of ports = 2
HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP
Fast-switchover     = disabled
Load share deferral = disabled

Ports in the Port-channel:

Index   Load      Port          EC state       No of bits
------+------+------------+------------------+-----------
0      55         Te10/2             Active   4
1      AA         Te10/3             Active   4

Time since last port bundled:    0d:05h:58m:32s    Te10/3
Time since last port Un-bundled: 0d:06h:00m:05s    Te10/3

Switch 2

Group: 2
----------
Group state = L2
Ports: 2   Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol:   LACP
Minimum Links: 0
                Ports in the group:
                -------------------
Port: Te9/2
------------

Port state    = Up Mstr In-Bndl
Channel group = 2           Mode = Active      Gcchange = -
Port-channel  = Po2         GC   =   -         Pseudo port-channel = Po2
Port index    = 0           Load = 0x55        Protocol =   LACP
Mode = LACP

Flags:  S - Device is sending Slow LACPDUs   F - Device is sending fast LACPDUs.
        A - Device is in active mode.        P - Device is in passive mode.

Local information:
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Te9/2     SA      bndl       32768         0x2       0x2     0x903       0x3D

Partner's information:

          Partner Partner   LACP Partner  Partner   Partner  Partner     Partner
Port      Flags   State     Port Priority Admin Key Oper Key Port Number Port State
Te9/2     SA      bndl       32768         0x0       0x3      0xA03       0x3D

Age of the port in the current state: 15d:05h:26m:56s

Port: Te9/3
------------

Port state    = Up Mstr In-Bndl
Channel group = 2           Mode = Active      Gcchange = -
Port-channel  = Po2         GC   =   -         Pseudo port-channel = Po2
Port index    = 1           Load = 0xAA        Protocol =   LACP
Mode = LACP

Flags:  S - Device is sending Slow LACPDUs   F - Device is sending fast LACPDUs.
        A - Device is in active mode.        P - Device is in passive mode.

Local information:
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Te9/3     SA      bndl       32768         0x2       0x2     0x904       0x3D

Partner's information:

          Partner Partner   LACP Partner  Partner   Partner  Partner     Partner
Port      Flags   State     Port Priority Admin Key Oper Key Port Number Port State
Te9/3     SA      bndl       32768         0x0       0x3      0xA04       0x3D

Age of the port in the current state: 0d:06h:00m:08s

                Port-channels in the group:
                ----------------------

Port-channel: Po2    (Primary Aggregator)

------------

Age of the Port-channel   = 15d:11h:25m:37s
Logical slot/port   = 14/1          Number of ports = 2
HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP
Fast-switchover     = disabled
Load share deferral = disabled

Ports in the Port-channel:

Index   Load      Port          EC state       No of bits
------+------+------------+------------------+-----------
0      55          Te9/2             Active   4
1      AA          Te9/3             Active   4

Time since last port bundled:    0d:06h:00m:10s    Te9/3
Time since last port Un-bundled: 0d:06h:01m:44s    Te9/3

4 Replies 4

mgalazka
Level 1
Level 1

Greetings.

Did either 6500's switch logs show anything?  5 min down unexpectedly, then re-establishing without any intervention sounds like a possible err-disable with recovery...

Matt

I turned up the logging last night and it happened again this morn Looks like Mod 9 card is resetting, any ideas this card has decided to start resetting itself ?

2w2d: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 9 to recover from error: Linecard received system exception. Errcode =
        2709000001
2w2d: %OIR-SP-3-PWRCYCLE: Card in module 9, is being power-cycled 'Off (Module Reset due to exception or user request)'
2w2d: %C6KPWR-SP-4-DISABLED: power to module in slot 9 set Off (Module Reset due to exception or user request)
*Mar  1 00:00:02.331: DaughterBoard (Centralized Forwarding Card)

Firmware compiled 15-Nov-11 14:23 by integ Build [100]
Feb 27 10:43:53.927: %SYS-CFC9-5-RESTART: System restarted --
Cisco IOS Software, c6lc2 Software (c6lc2-SP-M), Version 12.2(33)SXJ2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 14-Dec-11 19:50 by prod_rel_team
Feb 27 10:43:53.931: CFC9: Currently running ROMMON from S (Gold) region
2w2d: %DIAG-SP-6-RUN_MINIMUM: Module 9: Running Minimal Diagnostics...
2w2d: %DIAG-SP-6-DIAG_OK: Module 9: Passed Online Diagnostics
2w2d: %OIR-SP-6-INSCARD: Card inserted in slot 9, interfaces are now online

You can try reseating the module that is giving problems, or you can RMA it and get a replacement. Probably the easiest solution.

If you do get a replacement and it still resets itself then it could be a chassis issue but it is more likely to be the card.

Jon

2w2d: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 9 to recover from error: Linecard received system exception. Errcode =

        2709000001

Can you connect console to the line card and see if it is crashing?. Crash file generated can be helpful to debug. Meanwhile, getting replacement from TAC is best option.

Review Cisco Networking for a $25 gift card