GSR 12410 Active PRP-2 Crashed

[taimoor.jan]

Hi,

My GSR12410 Active PRP-2 crashed causing all services to resart. The Standby PRP-2 module took the control. Please suggest the root cause.

IOS: 12.0(33)S7

Loggs: soon after PRP module crashed:

++++++++++++++++++++++++++++++++++++++++++

000057: Mar 31 17:47:25.995 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up

000058: Mar 31 17:47:26.179 PST: %SYS-5-CONFIG_I: Configured from memory by console

000059: Mar 31 17:47:26.211 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1005, changed state to up

000060: Mar 31 17:47:26.211 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1605, changed state to up

000061: Mar 31 17:47:26.211 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback6005, changed state to up

000062: Mar 31 17:47:26.211 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback7001, changed state to up

000063: Mar 31 17:47:26.211 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback7002, changed state to up

000064: Mar 31 17:47:26.211 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback8001, changed state to up

000065: Mar 31 17:47:26.211 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback18005, changed state to up

000066: Mar 31 17:47:26.315 PST: %FPD_MGMT-3-INCOMP_IMG_VER: Incompatible CHOPPER PROM FP (FPD ID=1) image version detected for PRP-2 card in slot 8. Detected version = 2.149, minimum required version = 2.151. Current HW version = 10242.6.

000067: Mar 31 17:47:26.315 PST: %FPD_MGMT-4-UPGRADE_REQUIRED: FPD Image Upgrade is required for PRP-2 card in slot 8

000068: Mar 31 17:47:26.351 PST: %SYS-5-RESTART: System restarted --

Cisco Internetwork Operating System Software

IOS (tm) GS Software (C12KPRP-K4P-M), Version 12.0(33)S7, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by cisco Systems, Inc.

Compiled Mon 28-Jun-10 07:37 by debajpal

000069: Mar 31 17:47:26.351 PST: %SNMP-5-COLDSTART: SNMP agent on host LLR314-PE01 is undergoing a cold start

000070: Mar 31 17:47:26.423 PST: %SYS-6-BOOTTIME: Time taken to reboot after reload = 23385017 seconds

000071: Mar 31 17:47:26.643 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface POS1/0/3, changed state to down

000072: Mar 31 17:47:27.619 PST: %LINK-5-CHANGED: Interface POS1/0/1, changed state to administratively down

000073: Mar 31 17:47:27.683 PST: %LINK-5-CHANGED: Interface POS1/0/2, changed state to administratively down

000074: Mar 31 17:47:27.683 PST: %LINK-5-CHANGED: Interface POS1/0/4, changed state to administratively down

000075: Mar 31 17:47:27.683 PST: %LINK-5-CHANGED: Interface POS1/0/5, changed state to administratively down

000076: Mar 31 17:47:27.795 PST: %LINK-5-CHANGED: Interface Ethernet0, changed state to administratively down

000077: Mar 31 17:47:27.815 PST: %LINK-5-CHANGED: Interface Ethernet1, changed state to administratively down

000078: Mar 31 17:47:27.919 PST: %SSH-5-ENABLED: SSH 1.99 has been enabled

000079: Mar 31 17:47:28.618 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface POS1/0/1, changed state to down

000080: Mar 31 17:47:28.678 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface POS1/0/2, changed state to down

000081: Mar 31 17:47:28.682 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface POS1/0/4, changed state to down

000082: Mar 31 17:47:28.682 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface POS1/0/5, changed state to down

000083: Mar 31 17:47:28.794 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0, changed state to down

000084: Mar 31 17:47:28.814 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1, changed state to down

000085: Mar 31 17:47:29.266 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet2, changed state to down

000086: Mar 31 17:47:33.356 PST: %MBUS-6-FABCONFIG: Switch Cards 0x7F (bitmask)

Primary Clock is CSC_1

Fabric Clock is Redundant

Bandwidth Mode : 10Gbps Bandwidth

SLOT 1:00:01:42:  Setting LEDs

SLOT 1:00:01:42: %SYS-5-RESTART: System restarted --

Cisco Internetwork Operating System Software

IOS (tm) GS Software (GLC1-LC-M), Version 12.0(33)S7, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by cisco Systems, Inc.

Compiled Mon 28-Jun-10 07:21 by debajpal

SLOT 1:00:01:42: %LC-3-LAST_CRASH: LAST Shutdown Info: Reload from RP - interrupt:

SLOT 0:00:01:45: %SYS-5-RESTART: System restarted --

Cisco Internetwork Operating System Software

IOS (tm) GS Software (GLC1-LC-M), Version 12.0(33)S7, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by cisco Systems, Inc.

Compiled Mon 28-Jun-10 07:21 by debajpal

SLOT 0:00:01:45:  Setting LEDs

SLOT 0:Mar 31 17:49:16.255 PST: %LC-3-LAST_CRASH: LAST Shutdown Info: Reload from RP - interrupt:

00:00:03: %MBUS-6-FIA_CONFIG: Switch Cards 0x1F (bit mask); Primary Clock CSC_1

00:00:03: %MBUS-6-RP_STATUS: RP in Slot 9 Mode = MBUS Standby

000087: Mar 31 17:49:28.089 PST: %RP-6-REDMODE: Operating redundancy mode is RPR

000088: Mar 31 17:49:32.439 PST: %SONET-4-ALARM:  POS1/0/0: SLOS

000089: Mar 31 17:49:33.330 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface POS1/0/0, changed state to down

000090: Mar 31 17:49:35.317 PST: %RPGE-6-LINK_STATUS_DOWN: Interface GigabitEthernet0/0/1: Detected RX Link Status Down

000091: Mar 31 17:49:35.321 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/0/1: Link OK - autonegotiation complete

000092: Mar 31 17:49:35.329 PST: %SPA_OIR-6-ONLINECARD: SPA (SPA-2X1GE) online in subslot 0/0

000093: Mar 31 17:49:35.329 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/0/1: Link OK - autonegotiation complete

000094: Mar 31 17:49:35.789 PST: %RPGE-6-LINK_STATUS_DOWN: Interface GigabitEthernet0/1/0: Detected RX Link Status Down

000095: Mar 31 17:49:35.789 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/1/0: Link OK - autonegotiation complete

000096: Mar 31 17:49:35.885 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/1/0: Link OK - autonegotiation complete

000097: Mar 31 17:49:36.157 PST: %RP-3-RESTART: Linecard packet buffers recarved

000098: Mar 31 17:49:36.189 PST: %SPA_OIR-6-ONLINECARD: SPA (SPA-8XOC3-POS) online in subslot 1/0

000099: Mar 31 17:49:36.273 PST: %RPGE-6-LINK_STATUS_DOWN: Interface GigabitEthernet0/1/1: Detected RX Link Status Down

000100: Mar 31 17:49:36.281 PST: %SPA_OIR-6-ONLINECARD: SPA (SPA-2X1GE) online in subslot 0/1

000101: Mar 31 17:49:36.281 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/1/1: Link OK - autonegotiation complete

000102: Mar 31 17:49:36.281 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/1/1: Link OK - autonegotiation complete

000103: Mar 31 17:49:36.293 PST: %RPGE-6-LINK_STATUS_DOWN: Interface GigabitEthernet0/2/0: Detected RX Link Status Down

000104: Mar 31 17:49:36.293 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/2/0: Link OK - autonegotiation complete

000105: Mar 31 17:49:36.297 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/1, changed state to down

000106: Mar 31 17:49:36.305 PST: %SPA_OIR-6-ONLINECARD: SPA (SPA-2X1GE) online in subslot 0/2

000107: Mar 31 17:49:36.309 PST: %RPGE-6-LINK_STATUS_DOWN: Interface GigabitEthernet0/2/1: Detected RX Link Status Down

000108: Mar 31 17:49:36.309 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/2/1: Link OK - autonegotiation complete

000109: Mar 31 17:49:36.317 PST: %SONET-4-ALARM:  POS1/0/3: LRDI

000110: Mar 31 17:49:36.321 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/2/0: Link OK - autonegotiation complete

000111: Mar 31 17:49:36.337 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet0/2/1: Link OK - autonegotiation complete

000112: Mar 31 17:49:36.373 PST: %RPGE-6-LINE_STATE: Interface GigabitEthernet0/0/0: Line state OK

000113: Mar 31 17:49:36.377 PST: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/0, changed state to up

000114: Mar 31 17:49:36.445 PST: %SONET-4-ALARM:  POS1/0/6: LRDI

000115: Mar 31 17:49:36.449 PST: %LINK-3-UPDOWN: Interface POS1/0/6, changed state to up

000116: Mar 31 17:49:36.565 PST: %SONET-4-ALARM:  POS1/0/7: LRDI

000117: Mar 31 17:49:36.569 PST: %LINK-3-UPDOWN: Interface POS1/0/7, changed state to up

000118: Mar 31 17:49:36.589 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet1/1/0: Link OK - autonegotiation complete

000119: Mar 31 17:49:36.593 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet1/1/0: Link OK - autonegotiation complete

000120: Mar 31 17:49:36.601 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet1/1/1: Link OK - autonegotiation complete

000121: Mar 31 17:49:36.605 PST: %SPA_OIR-6-ONLINECARD: SPA (SPA-2X1GE-V2) online in subslot 1/1

000122: Mar 31 17:49:36.609 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet1/1/1: Link OK - autonegotiation complete

000123: Mar 31 17:49:36.625 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet1/1/0: Link OK - autonegotiation complete

000124: Mar 31 17:49:36.641 PST: %RPGE-6-AUTONEG_STATE: Interface GigabitEthernet1/1/1: Link OK - autonegotiation complete

000125: Mar 31 17:49:36.788 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1/0, changed state to down

000126: Mar 31 17:49:37.192 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface POS1/0/6, changed state to up

+++++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++++++++++++++++++++++++++++++++++++++

Please suggest the wayout and root cause.