CRS-1 Fabric Card Failure

[DavideL3]

Hello Everyone,
I have this problem with a stand-alone CRS-1.  The router boots up but it's unable to activate de fabric cards.
It's an old router with lastest PRP available  CRS-8-PRP-12G an XR version 6.1.2.
It was a spare part and it has been never used in a production environment

The fabric cards go int "IOS XR RUN" state but immedialy passes to "IOS XR FAILURE". 
The console shows this output: 

shelfmgrv2[404]: %PLATFORM-SHELFMGRV2-3-NODE_HBLOSS_RESET : Reset node 0/SM0/SP due to heartbeat loss
shelfmgrv2[404]: %PLATFORM-SHELFMGRV2-4-COLLECTING_DEBUGS_PRIOR_RESET : Timeout waiting for node 0/SM0/SP to transition into XR FAIL UNGRACEFUL state.

After 15 minute of trying the PRP reboots the entire chassis:
RP/0/RP0/CPU0:Nov 16 10:21:21.659 : fsdb_aserver[242]: %FABRIC-FSDB-1-FABRIC_UPDOWN : Fabric not up for data traffic
RP/0/RP0/CPU0:Nov 16 10:23:19.745 : sysmgr[94]: %OS-SYSMGR-2-REBOOT : reboot required, process (sysmgr) reason (Boot-hold by (fsdb_aserver) exceeded limit, timeout at 1202 seconds)

Could someone help me ?

Thank you

Davide

[smilstea · Accepted solution]

So the CRS-8 was never used before?

How many fabric cards are installed and in what slots?

Do all the fabric cards have a problem or only some of them fail?

There is a notorious problem on the 8-slot chassis that the fabric cards actually go in further than being flush with the rest of the chassis, if you are off by 1mm the cards will reboot themselves or observe other mysertious behavior. I would highly recommend pushing them in further, they might feel like they can't go in any further but they will and then you will see the difference between fabric card and chassis not lining up I am talking about. If thats not the case then some pictures and full logs would be helpful or a TAC SR if you feel more comfortable with that.

Sam

[DavideL3 · Accepted solution]

Hello Sam,
You got it! 
One FC card was not properly inserted into the slot (about 2mm off), so I pushed it further.
All FC cards now are in IOS XR RUN state and the fabric is up.

RP/0/RP0/CPU0:CRR01.PDVSIX(admin)#show plat
Sat Nov 17 12:47:52.017 UTC
Node Type PLIM State Config State
------------- ----------------- ------------------ --------------- ---------------
0/RP0/CPU0 RP(Active) N/A IOS XR RUN PWR,NSHUT,MON
0/RP1/CPU0 RP(Standby) N/A IOS XR RUN PWR,NSHUT,MON
0/SM0/SP FC-40G/S(SP) N/A IOS XR RUN PWR,NSHUT,MON
0/SM1/SP FC-40G/S(SP) N/A IOS XR RUN PWR,NSHUT,MON
0/SM2/SP FC-40G/S(SP) N/A IOS XR RUN PWR,NSHUT,MON
0/SM3/SP FC-40G/S(SP) N/A IOS XR RUN PWR,NSHUT,MON

Many thanks for the support!

Davide