Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
8054
Views
5
Helpful
12
Replies

ASR9K install add fails when 0/RSP1/CPU0 is in bringdown state

heleon
Cisco Employee
Cisco Employee

‎02-20-2017 04:18 PM

Hello Experts

I have attempted to install the sp4 (asr9k-px-5.3.3.sp4.tar)for IOS XR 5.3.3 in AR9006

However the install fails with a message about the state of 0/RSP1/CPU0.

Can I install the SP4 in spite of the status of 0/RSP1/CPU0?

Thanks in advance for your help below is the console log.

Thank you in advance for your help.

RP/0/RSP0/CPU0:ASR9006-2(admin)# install add disk0:asr9k-px-5.3.3.sp4.tar
Wed Feb 22 01:02:09.476 UTC
Install operation 20 '(admin) install add /disk0:asr9k-px-5.3.3.sp4.tar'
started by user 'root' via CLI at 01:02:09 UTC Wed Feb 22 2017.
RP/0/RSP0/CPU0:Feb 22 01:02:09.887 : instdir[254]: %INSTALL-INSTMGR-3-INSTALL_OPERATION_USER_ERROR : User error occurred during install operation 20. See 'show install log 20 detail' for more infError:    Cannot proceed with the add operation because the following node is
ormation.
Error:    not ready
Error:        0/RSP1/CPU0
Error:    Suggested steps to resolve this:
Error:     - repeat the operation once '(admin) show platform' shows the
Error:       specified node is in IOS-XR RUN state.
Install operation 20 failed at 01:02:09 UTC Wed Feb 22 2017.
RP/0/RSP0/CPU0:ASR9006-2(admin)#

Best regards

Hector Leon

Labels:
0 Helpful
12 Replies 12

smilstea
Cisco Employee
Cisco Employee

‎02-21-2017 12:10 AM

You won't be able to install with the router in this state.

Thanks,

Sam

0 Helpful

smailmilak
Level 4
Level 4

‎02-21-2017 12:19 AM

Hi Hector,

your priority should be RSP1 for now. Use these commands and give us the output.

1. admin show install log 20 detail

2. admin show platform sum loc 0/RSP1/CPU0

3. show redundancy

4. show environment leds location 0/RSP1/CPU0

0 Helpful

heleon
Cisco Employee
Cisco Employee
In response to smailmilak

‎02-21-2017 09:16 AM

Hi!

Thanks for your interest!

Below is the output for the commands

I will be paying attention to your comments.

Thanks in advance!

Best regards

Hector Leon

1. show install log 20 detail

RP/0/RSP0/CPU0:ASR9006-2#sh install log 20 detail

Wed Feb 22 17:47:42.075 UTC

 

Install operation 20 started by user 'root' via CLI at 01:02:09 UTC Wed Feb 22

2017.

(admin) install add /disk0:asr9k-px-5.3.3.sp4.tar

Install operation 20 failed at 01:02:09 UTC Wed Feb 22 2017.

 

Install logs:

    Install operation 20 '(admin) install add /disk0:asr9k-px-5.3.3.sp4.tar'

    started by user 'root' via CLI at 01:02:09 UTC Wed Feb 22 2017.

    Error:    Cannot proceed with the add operation because the following node

    Error:    is not ready

    Error:        0/RSP1/CPU0

    Error:    Suggested steps to resolve this:

    Error:     - repeat the operation once '(admin) show platform' shows the

    Error:       specified node is in IOS-XR RUN state.

    Install operation 20 failed at 01:02:09 UTC Wed Feb 22 2017.

 

--------------------------------------------------------------------------------

 

RP/0/RSP0/CPU0:ASR9006-2#RP/0/RSP0/CPU0:Feb 22 17:47:45.514 : devc-conaux[59]: %MGBL-ST16550-3-ISR_MSR_ERR : Too many console line state change interrupts

2.admin show platform sum loc 0/RSP1/CPU0

RP/0/RSP0/CPU0:ASR9006-2#RP/0/RSP0/CPU0:Feb 22 17:47:45.514 : devc-conaux[59]: %MGBL-ST16550-3-ISR_MSR_ERR : Too many console line state change interrupts

 

RP/0/RSP0/CPU0:ASR9006-2#admin show platform sum loc 0/RSP1/CPU0

Wed Feb 22 17:48:51.625 UTC

-------------------------------------------------------------------------------

     Platform Node : 0/RSP1/CPU0 (slot 1)

               PID : A9K-RSP440-SE

         Card Type : ASR9K Fabric, Controller, 12G memory

            VID/SN : V01 / FOC160785Q6

        Oper State : BRINGDOWN

        Last Reset : N/A

                   : N/A

     Configuration : Power is enabled

                     Bootup enabled.

                     Monitoring enabled

        Rommon Ver : N/A

        IOS SW Ver : 5.3.3

        Main Power : Power state Enabled. Estimate power 350 Watts of power required.

            Faults : N/A

-------------------------------------------------------------------------------

RP/0/RSP0/CPU0:ASR9006-2#RP/0/RSP0/CPU0:Feb 22 17:48:55.239 : devc-conaux[59]: %MGBL-ST16550-3-ISR_MSR_ERR : Too many console line state change interrupts

 

 3 .show redundancy

RP/0/RSP0/CPU0:ASR9006-2#show redundancy

Wed Feb 22 17:49:26.547 UTC

Redundancy information for node 0/RSP0/CPU0:

==========================================

Node 0/RSP0/CPU0 is in ACTIVE role

Node 0/RSP0/CPU0 has no valid partner

 

Group            Primary         Backup          Status

---------        ---------       ---------       ---------

dsc              0/RSP0/CPU0     N/A             Not Ready

dlrsc            0/RSP0/CPU0     N/A             Not Ready

central-services 0/RSP0/CPU0     N/A             Not Ready

v4-routing       0/RSP0/CPU0     N/A             Not Ready

netmgmt          0/RSP0/CPU0     N/A             Not Ready

mcast-routing    0/RSP0/CPU0     N/A             Not Ready

mcast-routing    0/RSP0/CPU0     N/A             Not NSR-Ready

v6-routing       0/RSP0/CPU0     N/A             Not Ready

 

Process Group Details

---------------------

 

Current primary rmf state: Not Ready

  <jid>       <node>       <name>      <group> Reason for backup not ready

    398  0/RSP0/CPU0      rmf_svr          dsc No location for backup set

        Not ready set Sat Feb  4 21:42:44 2017: 2 weeks, 3 days, 20 hours, 6 minutes ago

    398  0/RSP0/CPU0      rmf_svr        dlrsc No location for backup set

        Not ready set Sat Feb  4 21:42:45 2017: 2 weeks, 3 days, 20 hours, 6 minutes ago

    398  0/RSP0/CPU0      rmf_svr central-services No location for backup set

        Not ready set Sat Feb  4 21:43:41 2017: 2 weeks, 3 days, 20 hours, 5 minutes ago

    398  0/RSP0/CPU0      rmf_svr   v4-routing No location for backup set

        Not ready set Sat Feb  4 21:43:41 2017: 2 weeks, 3 days, 20 hours, 5 minutes ago

    398  0/RSP0/CPU0      rmf_svr      netmgmt No location for backup set

        Not ready set Sat Feb  4 21:43:41 2017: 2 weeks, 3 days, 20 hours, 5 minutes ago

    398  0/RSP0/CPU0      rmf_svr mcast-routing No location for backup set

        Not ready set Sat Feb  4 21:43:41 2017: 2 weeks, 3 days, 20 hours, 5 minutes ago

    398  0/RSP0/CPU0      rmf_svr   v6-routing No location for backup set

        Not ready set Sat Feb  4 21:43:41 2017: 2 weeks, 3 days, 20 hours, 5 minutes ago

 

Current primary rmf state for NSR: Not Ready

  <jid>       <node>       <name>      <group> Reason for backup not NSR-ready

   1169  0/RSP0/CPU0         igmp mcast-routing Standby process not converged

       Not ready set Sat Feb  4 21:43:46 2017: 2 weeks, 3 days, 20 hours, 5 minutes ago

   1174  0/RSP0/CPU0          mld mcast-routing Standby process not converged

        Not ready set Sat Feb  4 21:43:46 2017: 2 weeks, 3 days, 20 hours, 5 minutes ago

 

Reload and boot info

----------------------

A9K-RSP440-SE reloaded Sat Feb  4 21:41:44 2017: 2 weeks, 3 days, 20 hours, 7 minutes ago

Active node booted Sat Feb  4 21:41:44 2017: 2 weeks, 3 days, 20 hours, 7 minutes ago

 

Active node reload "Cause: User Initiated reload"

 

RP/0/RSP0/CPU0:ASR9006-2#RP/0/RSP0/CPU0:Feb 22 17:49:30.336 : devc-conaux[59]: %MGBL-ST16550-3-ISR_MSR_ERR : Too many console line state change interrupts

 

 

4. show environment leds location 0/RSP1/CPU0

RP/0/RSP0/CPU0:ASR9006-2#RP/0/RSP0/CPU0:Feb 22 17:49:30.336 : devc-conaux[59]: %MGBL-ST16550-3-ISR_MSR_ERR : Too many console line state change interrupts

 

RP/0/RSP0/CPU0:ASR9006-2#show environment leds location 0/RSP1/CPU0

Wed Feb 22 17:50:05.661 UTC

R/S/I   Modules LED             Status

0/RSP1/*

        host    Critical-Alarm  Off

        host    Major-Alarm     Off

        host    Minor-Alarm     Off

        host    ACO             Off

        host    Fail            Off

RP/0/RSP0/CPU0:ASR9006-2#

0 Helpful

smailmilak
Level 4
Level 4
In response to heleon

‎02-21-2017 09:43 AM

Sorry for not noticing that the discussion has "bringdown" in the title. 

Give us the output of show logging.

You have a serious issue with RSP1. You could try to re-insert it or maybe

do admin conf ter hw-module power disable location 0/RSP1/CPU0

commit

and then enable power again.

Copy the logs when you do this and paste it here.

It is possible that you need to replace the RSP module.

0 Helpful

heleon
Cisco Employee
Cisco Employee
In response to smailmilak

‎02-21-2017 11:02 AM

Hi!

Thank you for your comments we attempted power disable and IOS XR indicates :

Stop is not supported on this node.

Reloading the card indicates failure and then bringdown.

We will re-seat the line card. 

RP/0/RSP0/CPU0:ASR9006-2(admin)#hw-module location 0/RSP1/CPU0 stop

Wed Feb 22 19:41:47.963 UTC

WARNING: This will take the requested node out of service.

Do you wish to continue? [confirm(y/n)]y

Stop is not supported on this node.

RP/0/RSP0/CPU0:ASR9006-2(admin)#show platform

Wed Feb 22 19:42:01.828 UTC

Node            Type                      State            Config State

-----------------------------------------------------------------------------

0/RSP0/CPU0     A9K-RSP440-SE(Active)     IOS XR RUN       PWR,NSHUT,MON

0/RSP1/CPU0     A9K-RSP440-SE(Standby)    BRINGDOWN        PWR,NSHUT,MON

0/FT0/SP        ASR-9006-FAN              READY

0/FT1/SP        ASR-9006-FAN              READY

0/0/CPU0        A9K-24x10GE-SE            IOS XR RUN       PWR,NSHUT,MON

0/1/CPU0        A9K-MOD160-SE             IOS XR RUN       PWR,NSHUT,MON

0/1/0           A9K-MPA-4X10GE            OK               PWR,NSHUT,MON

0/2/CPU0        A9K-2T20GE-B              IOS XR RUN       PWR,NSHUT,MON

0/3/CPU0        A9K-MOD80-TR              IOS XR RUN       PWR,NSHUT,MON

0/3/1           A9K-MPA-20X1GE            OK               PWR,NSHUT,MON

0/PM0/0/SP      A9K-3KW-AC                FAILED           PWR,NSHUT,MON

0/PM0/1/SP      A9K-3KW-AC                READY            PWR,NSHUT,MON

0/PM0/2/SP      A9K-3KW-AC                READY            PWR,NSHUT,MON

0 Helpful

smailmilak
Level 4
Level 4
In response to heleon

‎02-21-2017 11:51 PM

Yeah right, it's not possible to disable power on RSP or RP. 

Let us know after you are done with re-insert of RSP1 and give us the logs.

Do you have noticed that power supply in PM0/0 has failed?

0 Helpful

heleon
Cisco Employee
Cisco Employee
In response to smailmilak

‎02-22-2017 12:31 PM

Hello Team

I used Cisco power calculator and two Power Supplies with 3000W it should be enough.

Below the logs after re-seat the RSP in bringdown.

Thanks in advance.

RP/0/RSP0/CPU0:Feb 23 19:47:32.932 : shelfmgr[410]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:PRESENT
RP/0/RSP0/CPU0:Feb 23 19:47:33.003 : invmgr[257]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/RSP1/CPU0 Sn: FOC160785Q6 inserted 
RP/0/RSP0/CPU0:Feb 23 19:50:02.421 : shelfmgr[410]: %PLATFORM-SHELFMGR-3-FSMTIMEOUT_RESET : Node 0/RSP1/CPU0 is reset due to failed bootup. Node state was: 1 Timeout ID: 10
RP/0/RSP0/CPU0:Feb 23 19:50:02.439 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , Power Cycle (0x05000000) 
RP/0/RSP0/CPU0:Feb 23 19:50:02.440 : shelfmgr[410]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:ROMMON
RP/0/RSP0/CPU0:Feb 23 19:52:24.905 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , DB Power Failure (0x0f000000) 
RP/0/RSP0/CPU0:Feb 23 19:52:24.905 : canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_RSP_DB_PWR_FAILED 
RP/0/RSP0/CPU0:Feb 23 19:52:24.906 : shelfmgr[410]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:BRINGDOWN
RP/0/RSP0/CPU0:Feb 23 19:52:24.908 : invmgr[257]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/RSP1/CPU0, state: BRINGDOWN
RP/0/RSP0/CPU0:Feb 23 19:52:52.550 : canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_CLEARED : Clear alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_RSP_DB_PWR_FAILED 
RP/0/RSP0/CPU0:Feb 23 19:54:54.962 : shelfmgr[410]: %PLATFORM-SHELFMGR-3-FSMTIMEOUT_RESET : Node 0/RSP1/CPU0 is reset due to failed bootup. Node state was: 7 Timeout ID: 10
RP/0/RSP0/CPU0:Feb 23 19:54:54.991 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , Power Cycle (0x05000000) 
RP/0/RSP0/CPU0:Feb 23 19:57:19.101 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , DB Power Failure (0x0f000000) 
RP/0/RSP0/CPU0:Feb 23 19:57:19.101 : canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_RSP_DB_PWR_FAILED 
RP/0/RSP0/CPU0:Feb 23 19:57:58.731 : canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_CLEARED : Clear alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_RSP_DB_PWR_FAILED 
RP/0/RSP0/CPU0:Feb 23 20:27:20.854 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , WDOG SReset (0x06000000) 
RP/0/RSP0/CPU0:Feb 23 20:27:36.895 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , WDOG HReset (0x07000000) 
RP/0/RSP0/CPU0:Feb 23 20:27:37.891 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , WDOG Power Cycle (0x08000000) 
RP/0/RSP0/CPU0:Feb 23 20:30:07.367 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , DB Power Failure (0x0f000000) 
RP/0/RSP0/CPU0:Feb 23 20:30:07.367 : canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_RSP_DB_PWR_FAILED 
RP/0/RSP0/CPU0:Feb 23 20:30:33.007 : canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_CLEARED : Clear alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_RSP_DB_PWR_FAILED 
RP/0/RSP0/CPU0:Feb 23 21:02:51.636 : canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_RSP_DB_PWR_FAILED 
 RP/0/RSP0/CPU0:Feb 23 21:03:07.270 : canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_CLEARED : Clear alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_RSP_DB_PWR_FAILED 

0 Helpful

heleon
Cisco Employee
Cisco Employee
In response to heleon

‎02-22-2017 03:42 PM

We will replace the RSP1

Best regards

0 Helpful

smailmilak
Level 4
Level 4
In response to heleon

‎02-23-2017 12:05 AM

It's clearly a case for Cisco TAC.

Please check also the failed power supply. 

And btw. I see that you want to activate SP4 for 5.3.3. 

Our recommendation is to activate 5.3.4 and SP1 for this version (if it's available)!!!

You can do it now, just remove RSP1 from the chassis and the activation process should be ok.

0 Helpful

heleon
Cisco Employee
Cisco Employee
In response to smailmilak

‎02-23-2017 08:51 AM

Thank You!

We removed the RSP1 and installed the PIE and SP.

Also we requested the replace and we will  install the new RSP1 as soon as possible.

Thank you again for your valuable help and comments.

Best regards

0 Helpful

Pichet Piriyakitkosol
Level 1
Level 1
In response to smailmilak

‎02-03-2019 08:25 PM

Can we do "hw-module location 0/rsp1/CPU0 reload" instead of remove and re-insert the standby RSP?

 

Thank you

0 Helpful

smilstea
Cisco Employee
Cisco Employee
In response to Pichet Piriyakitkosol

‎02-05-2019 05:39 AM

Yes you can reload the RSP via that command, the thing you can't do is power disable a RSP.

 

Sam

Customers Also Viewed These Support Documents