Solved: ISM card stuck in MBI-running - Page 2

cisco_lad2004 · ‎09-27-2012

Dear all,

I am trying to bring up an ISM card installed on ASR9K running 4.2.1.

Card shows under inventory, but keeps pop in in and out of "show platform" as well between below 2 states:

0/2/CPU0 A9K-ISM-100(LCP) IOS XR PREP PWR,NSHUT,MON

And

0/2/CPU0 A9K-ISM-100(LCP) MBI-RUNNING PWR,NSHUT,MON

RP/0/RSP0/CPU0:CGN_PE#sh hw-module fpd location 0/2/CPU0

Wed Sep 26 16:13:04.966 CETDST

===================================== ==========================================

Existing Field Programmable Devices

==========================================

HW Current SW Upg/

Location Card Type Version Type Subtype Inst Version Dng?

============ ======================== ======= ==== ======= ==== =========== ====

0/2/CPU0 does not exist

I tried reloading card few times after following procedures for fpd upgrade. I am beginiing to suspect fpd is corrupted.

Any thoughts ?

TIA

/Samir

cisco_lad2004 · ‎09-27-2012

Will do, many thanks !

so thsi message, so tried below with no joy !

proc mandatory OFF fib_mgr location 0/2/CPU0

proc SHUTDOWN fib_mgr location 0/2/CPU0

RP/0/RSP0/CPU0:#admin

debug sim reload-disable location 0/2/CPU0

cisco_lad2004 · ‎09-27-2012

well...something has worked:

RP/0/RSP0/CPU0:CGN_PE#sh platform

Thu Sep 27 16:40:38.260 CETDST

Node Type State Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0 A9K-RSP440-SE(Active) IOS XR RUN PWR,NSHUT,MON

0/0/CPU0 A9K-MOD80-TR IOS XR RUN PWR,NSHUT,MON

0/0/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON

0/2/CPU0 A9K-ISM-100(LCP) IOS XR RUN PWR,NSHUT,MON

0/2/CPU1 A9K-ISM-100(SE) FAILED

Alexei Kiritchenko · ‎09-27-2012

Yes, after your shut down the fib manager on the card it does not reload the ISM due to the version incompatibility.

you may update PFD so far in order to prepare for the kit installation

(admin)#upgrade hw-module fpd all location 0/2/CPU0

Regards,

/A

cisco_lad2004 · ‎09-27-2012

Thats right !

I did that already, then reloaded 0/2/CPU0 and it is still holding...so looking good so far.

Thanks again !

/Samir

cisco_lad2004 · ‎09-28-2012

TAC request sent, with subject :ISM card stuck in MBI-running. Cisco IOS XR 4.2.1 CGN

Alexei Kiritchenko · ‎09-30-2012

Summary:

The image has been provided to Sam but we could not deploy it.

We have found the CGN installation kit was corrupted. After re-downloading the package we could successfully deploy it.

Note: the image itself is a text script with binary together and starts as a text file. We suspect, FTP was detecting the image as TXT and hence working in text mode replacing new line and carrier return characters. Forcing FTP into binary mode we could successfully download it.

Advise is to verify MD5 checksum for downloaded SW

cisco_lad2004 · ‎09-30-2012

Thanks a million Alex, glad card is now ready to do what is supposed to. And yes, great advice, it could have saved me loads of valuable time :-)

Renato Reis · ‎11-01-2013

Hello Samir,

I have a ISM module that cant come online, could help me with that? The problem happened before the ASR reloaded.

RP/0/RSP1/CPU0:router#admin show platform

Fri Nov 1 16:17:16.265 UTC

Node Type State Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0 A9K-RSP440-TR(Standby) IOS XR RUN PWR,NSHUT,MON

0/RSP1/CPU0 A9K-RSP440-TR(Active) IOS XR RUN PWR,NSHUT,MON

0/FT0/SP FAN TRAY READY

0/FT1/SP FAN TRAY READY

0/0/CPU0 A9K-MOD160-TR IOS XR RUN PWR,NSHUT,MON

0/0/0 A9K-MPA-8X10GE OK PWR,NSHUT,MON

0/0/1 A9K-MPA-8X10GE OK PWR,NSHUT,MON

0/1/CPU0 A9K-MOD160-TR IOS XR RUN PWR,NSHUT,MON

0/1/0 A9K-MPA-8X10GE OK PWR,NSHUT,MON

0/1/1 A9K-MPA-8X10GE OK PWR,NSHUT,MON

0/3/CPU0 A9K-ISM-100(LCP) IN-RESET PWR,NSHUT,MON

0/3/SAM0 A9K-SAM-2TB BOOTING PWR,NSHUT,NMON

0/3/SAM1 A9K-SAM-2TB FAILED PWR,NSHUT,NMON

0/PM0/SP PWR-3KW-AC-V2 READY PWR,NSHUT,MON

0/PM1/SP PWR-3KW-AC-V2 READY PWR,NSHUT,MON

0/PM2/SP PWR-3KW-AC-V2 READY PWR,NSHUT,MON

RP/0/RSP1/CPU0:router#

LC/0/3/CPU0:Nov 1 16:16:11.475 : apiv_srvr[115]: %PLATFORM-APIV-3-MODULE_INCOMPATIBLE : APIV module APIV_SI is incompatible

LC/0/3/CPU0:Nov 1 16:16:11.697 : vkg_l2fib_mac_cache[340]: %PLATFORM-PLAT_L2FIB-6-GENERIC : avsm l2fib: initializing L2 FIB MAC CAHE APIV modules.

LC/0/3/CPU0:Nov 1 16:16:11.705 : /pkg/sbin/sysmgr_log[65661]: %OS-SYSMGR-7-CHECK_LOG : /pkg/bin/sysmgr_debug_script invoked for: (fib_mgr) process did not signal EOI and for level:100 . Output is in harddisk:/sysmgr_debug/debug.node0_3_CPU0.442485

LC/0/3/CPU0:Nov 1 16:16:11.801 : vkg_l2fib_mac_cache[340]: %PLATFORM-PLAT_L2FIB-6-GENERIC : avsm l2fib: retrieving SysDB notifications ...

LC/0/3/CPU0:Nov 1 16:16:11.801 : vkg_l2fib_mac_cache[340]: %PLATFORM-PLAT_L2FIB-6-GENERIC : avsm l2fib: Starting of L2FIB MAC CACHE APIV version negotiation.

LC/0/3/CPU0:Nov 1 16:16:13.624 : canb_upg_agt[132]: %PLATFORM-UPGRADE_FPD-4-DOWN_REV : cbc instance 0 is down-rev (V18.05), upgrade to (V18.08). Use the "upgrade hw-module fpd" CLI in admin mode.

LC/0/3/CPU0:Nov 1 16:16:14.119 : l2fib[239]: %PLATFORM-PLAT_L2FIB-6-GENERIC : avsm l2fib: Initializing L2 FIB APIV modules.

LC/0/3/CPU0:Nov 1 16:16:14.192 : l2fib[239]: %PLATFORM-PLAT_L2FIB-6-GENERIC : avsm l2fib: retrieving SysDB notifications.

LC/0/3/CPU0:Nov 1 16:16:14.192 : l2fib[239]: %PLATFORM-PLAT_L2FIB-6-GENERIC : avsm l2fib: starting of APIV version negotiation.

LC/0/3/CPU0:Nov 1 16:16:14.784 : avsm_se_sm[125]: %PLATFORM-SIM_SE-6-STATE_CHANGE : Service Engine is in shutting down state.

LC/0/3/CPU0:Nov 1 16:16:14.790 : sis[307]: %PLATFORM-VKG_SIS-6-GENERIC_EVENT : sis event: Starting of SI APIV Deletion ...

LC/0/3/CPU0:Nov 1 16:16:15.420 : apiv_srvr[115]: %PLATFORM-APIV-5-API_INCOMPATIBLE : Message 8 in APIV module ISM_L2FIB_APIV is incompatible (reason: Not supported on the remote side)

LC/0/3/CPU0:Nov 1 16:16:15.420 : apiv_srvr[115]: %PLATFORM-APIV-3-MODULE_INCOMPATIBLE : APIV module ISM_L2FIB_APIV is incompatible

LC/0/3/CPU0:Nov 1 16:16:16.421 : avsm_agent[124]: %ISM_L2_UFA-3-ERR : L2UFA: apiv/ipcp init failed

LC/0/3/CPU0:Nov 1 16:16:21.641 : avsm_agent[124]: %SAM_OIR-3-SAM_ERR_FPD : SAM/1: Unable to initialize. Error in FPD of device(s):CPLD. Detailed message -23:Unable to connect to socket:/tmp/fpdinfra_socket -23 111:Connection refused

RP/0/RSP1/CPU0:Nov 1 16:16:21.647 : invmgr[257]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/3/SAM1, state: FAILED

LC/0/3/CPU0:Nov 1 16:16:24.704 : fib_mgr[172]: %PLATFORM-PLAT_FIB-3-ERR_STR_NEGO_FAIL : CGN fib apiv ipcp init: Failed to negotiate the apiv version.: UNKNOWN (12293)

LC/0/3/CPU0:Nov 1 16:16:30.695 : fib_mgr[172]: %ROUTING-FIB-3-TABLE : table operation failed during Invalid table create failed: Resource temporarily unavailable : pkg/bin/fib_mgr : (PID=434279) : -Traceback= 4002b9cc 4001426c 40014630 400019f4 4ba73a44 4ba71554 400003f0 40014a78 4bada400

RP/0/RSP1/CPU0:Nov 1 16:16:37.142 : shelfmgr[387]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/3/CPU0 A9K-ISM-100 state:IOS XR RUN

RP/0/RSP1/CPU0:Nov 1 16:16:37.144 : invmgr[257]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/3/CPU0, state: IOS XR RUN

LC/0/3/CPU0:Nov 1 16:16:47.370 : avsm_se_sm[125]: %PLATFORM-SIM_SE-6-STATE_CHANGE : Service Engine is in reset(Service Engine Software) state.

RP/0/RSP0/CPU0:Nov 1 16:16:47.418 : canb-server[149]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/3/CPU0 , Power Cycle (0x05000000)

RP/0/RSP0/CPU0:Nov 1 16:16:47.418 : canb-server[149]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/3/CPU0 , PLDREQ Sreset (0x0b000000)

RP/0/RSP1/CPU0:Nov 1 16:16:47.424 : canb-server[149]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/3/CPU0 , Power Cycle (0x05000000)

RP/0/RSP1/CPU0:Nov 1 16:16:47.424 : canb-server[149]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/3/CPU0 , PLDREQ Sreset (0x0b000000)

RP/0/RSP1/CPU0:Nov 1 16:16:47.425 : shelfmgr[387]: %PLATFORM-SHELFMGR-0-MAX_RESET_BRINGDOWN : Can not boot node 0/3/CPU0 A9K-ISM-100 due to multiple resets, putting it IN_RESET state. The probable cause is an unexpected event on the node or a failure in communication with the node. Please refer to the Cisco ASR 9000 System Error Message Reference Guide for further information if needed.

RP/0/RSP1/CPU0:Nov 1 16:16:47.425 : shelfmgr[387]: %PLATFORM-SHELFMGR-6-NODE_CPU_RESET : Node 0/3/CPU0 CPU reset detected.

RP/0/RSP1/CPU0:Nov 1 16:16:47.427 : shelfmgr[387]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/3/CPU0 A9K-ISM-100 state:IN-RESET

RP/0/RSP1/CPU0:Nov 1 16:16:47.434 : mibd_entity[324]: %HA-HA_EM-7-FMFD_CONNECTION_FAIL : Could not connect to /dev/fm/fd_wdsysmon.d/node0_3_CPU0 : No such file or directory

RP/0/RSP0/CPU0:Nov 1 16:16:47.548 : ce_switch_srv[53]: %PLATFORM-CE_SWITCH-6-UPDN : Interface 7 (LC_Slot_3) is down

RP/0/RSP1/CPU0:Nov 1 16:16:47.683 : ce_switch_srv[53]: %PLATFORM-CE_SWITCH-6-UPDN : Interface 7 (LC_Slot_3) is down

RP/0/RSP1/CPU0:Nov 1 16:16:48.436 : mibd_entity[324]: %HA-HA_EM-7-FMFD_CONNECTION_FAIL : Could not connect to /dev/fm/fd_wdsysmon.d/node0_3_CPU0 : No such file or directory

RP/0/RSP1/CPU0:Nov 1 16:16:50.437 : mibd_entity[324]: %HA-HA_EM-7-FMFD_CONNECTION_FAIL : Could not connect to /dev/fm/fd_wdsysmon.d/node0_3_CPU0 : No such file or directory

RP/0/RSP1/CPU0:Nov 1 16:16:54.439 : mibd_entity[324]: %HA-HA_EM-7-FMFD_CONNECTION_FAIL : Could not connect to /dev/fm/fd_wdsysmon.d/node0_3_CPU0 : No such file or directory

RP/0/RSP1/CPU0:router#

Thanks in advance,

Renato