Re: NCS5508 - RP confusion and a case of mistaken RP identity ?

Mark Pace Balzan · ‎07-30-2021

Hi,

I need some help from some cisco folk around about a possible issue im seeing on NCS5508

RP0 is apparently the active RP as can be seen in the below and things mostly work.

Except that sometimes the box claims RP0 is standby (but its lying)

(eg when i try to attach to a fabric card - granted you dont need this every day, but if needed its not possible to execute)

outputs below show better the situation im facing

box is running 7.2.2, issue was also present on 7.1

other 5508 box with identical hardware and software does not have this issue, so something wrong is happening here,

would appreciate if anyone can give any insight on this

tac case open but no joy yet

thanks in advance for you help

Mark

RP/0/RP0/CPU0:NCS5508-1#show redundancy
Thu Jul 22 08:05:15.241 CEST
Redundancy information for node 0/RP0/CPU0:
==========================================
Node 0/RP0/CPU0 is in ACTIVE role
Partner node (0/RP1/CPU0) is in STANDBY role
Standby node in 0/RP1/CPU0 is ready
Standby node in 0/RP1/CPU0 is NSR-ready




RP/0/RP0/CPU0:NCS5508-1#show platform
Thu Jul 22 08:05:31.766 CEST
Node Type State Config state
--------------------------------------------------------------------------------
0/RP0/CPU0 NC55-RP-E(Active) IOS XR RUN NSHUT
0/RP1/CPU0 NC55-RP-E(Standby) IOS XR RUN NSHUT




RP/0/RP0/CPU0:NCS5508-1#admin show controller card-mgr inventory summary
Thu Jul 22 08:05:57.344 CEST
Card Manager Inventory Summary :
BP HW
Location Card Type ID Serial Number Ver Card State
------------------------------------------------------------------------------
0/RP0 NC55-RP-E (Slave) 27 JAE245000A0 1.1 CARD_READY <<<<<======== RP0 is the real Active !
0/RP1 NC55-RP-E (Master) 28 JAE245000CF 1.1 CARD_READY




[sysadmin-vm:0_RP0:~]$.attach_console 25
SLAVE-RP: Please run Remote console connection from Master RP <<<<<====== Same here. I am connected to the active RP, otherwise I wouldnt even have CLI access !!

tkarnani · ‎07-30-2021

Hi Mark,

there are two containers, the admin and XR vm.

the admin container can run on either of the RP's

i have attached output from our lab.

admin container is active on RP1, xr container active on RP0

you can type "admin" hit enter to go to the container and check if the prompt changes to 0_RP1

sysadmin-vm:0_RP1# show plat | inc RP
Fri Jul  30 12:48:58.358 UTC+00:00
0/RP0     NC55-RP                 OPERATIONAL   OPERATIONAL   NSHUT
0/RP1     NC55-RP-E               OPERATIONAL   OPERATIONAL   NSHUT
sysadmin-vm:0_RP1# exit
Fri Jul  30 12:49:02.330 UTC+00:00
show pRP/0/RP0/CPU0:NCS-5508-B#show plat | inc RP
Fri Jul 30 04:49:07.353 GMT+8
0/RP0/CPU0        NC55-RP(Active)            IOS XR RUN        NSHUT
0/RP1/CPU0        NC55-RP-E(Standby)         IOS XR RUN        NSHUT
RP/0/RP0/CPU0:NCS-5508-B#

Mark Pace Balzan · ‎07-30-2021

Hi,

thanks for your reply

actually, if I enter admin, the prompt remains as RP0

RP/0/RP0/CPU0:NCS5508-1#admin
Fri Jul 30 14:55:17.242 CEST
Last login: Thu Jul 29 14:07:50 2021 from 192.0.108.4

admin connected from 192.0.108.4 using ssh on sysadmin-vm:0_RP0
sysadmin-vm:0_RP0#

even if then it shows up as RP1 in

admin show controller card-mgr inventory summary

and attempting to attach to the fabric is not possible as it 'complains' im on the slave RP (which is not the case)

[sysadmin-vm:0_RP0:~]$.attach_console 25
SLAVE-RP: Please run Remote console connection from Master RP

So the questions are:

- I understand from you it appears to be normal that the sysadmin plane may be running on RP1 and XR plane on RP0. Thats good to know I wasnt aware.

- However, I dont think this is what we are seeing here right ? Both are apparently running on RP0, and I cant seem to attach. I dont think this is expected behaviour ?

thanks

Mark

mivens · ‎08-09-2021

There are three separate, unrelated things:

For the XR VM on each RP, which is active.

For the admin VM on each RP which has the active confd_helper role and which is the slave. This is the RP that shows up in the command prompt when you type "admin" and you can see more information in the output of "show process confd_helper location all":

sysadmin-vm:0_RP0# show processes confd_helper location all
Mon Aug 9 18:08:27.100 UTC+01:00
----------------------------------------------------------------------
node: 0/RP0
----------------------------------------------------------------------
PID: 3368
Executable path: /opt/cisco/calvados/packages/ncs5500-sysadmin-mgbl-7.1.2-r712/rp/bin/confd_helper
Instance #: 0
Args: -t token -d -w 400 -b 30 -p 600 -r 10 -f 10
Respawn: ON
Respawn count: 1
Max. spawns per 4 mins: 4
Last started: 07/14/2021 00:29:10.000
Process state: Run
startup_path: /opt/cisco/calvados/packages/ncs5500-sysadmin-mgbl-7.1.2-r712/rp/etc/startup/confd_helper.startup
Ready: 0s
Table of services hosted on this process:
----------------------------------------------------------------------
LAST STARTED SCOPE SELE- ROLE STATE HA- SERVICE NAME
CTED RDY
----------------------------------------------------------------------
07/14/2021 00:30:02.000 SYS Y ACT Run confd_helper

----------------------------------------------------------------------

----------------------------------------------------------------------
node: 0/RP1
----------------------------------------------------------------------
PID: 3479
Executable path: /opt/cisco/calvados/packages/ncs5500-sysadmin-mgbl-7.1.2-r712/rp/bin/confd_helper
Instance #: 0
Args: -t token -d -w 400 -b 30 -p 600 -r 10 -f 10
Respawn: ON
Respawn count: 1
Max. spawns per 4 mins: 4
Last started: 07/13/2021 23:29:08.000
Process state: Run
startup_path: /opt/cisco/calvados/packages/ncs5500-sysadmin-mgbl-7.1.2-r712/rp/etc/startup/confd_helper.startup
Ready: 0s
Table of services hosted on this process:
----------------------------------------------------------------------
LAST STARTED SCOPE SELE- ROLE STATE HA- SERVICE NAME
CTED RDY
----------------------------------------------------------------------
07/13/2021 23:30:02.000 SYS Y STBY Run Y confd_helper

----------------------------------------------------------------------

If from admin you kill the active confd_helper process you will get kicked out back to XR and if you issue the admin command again you will find you are on the other RP and its confd_helper process is now active.

Lastly there is which RP has mastership for the card_mgr process and that's in the output of "show controller card-mgr inventory" as you pointed out earlier. If you want to attach to a console, you need to do that from the master, as the output from the .attach_console or rconsole commands tells you.

E.g.:


sysadmin-vm:0_RP0# show controller card-mgr inventory summary | include RP
Mon Aug 9 18:19:24.460 UTC+01:00
0/RP0 NC55-RP-E (Slave) 27 JAExxxxxxxxxD 1.1 CARD_READY
0/RP1 NC55-RP-E (Master) 28 JAExxxxxxxxF 1.1 CARD_READY

sysadmin-vm:0_RP0# run /opt/cisco/calvados/sbin/rconsole -l 0/4
Mon Aug 9 18:20:01.120 UTC+01:00
Connecting to Card Manager Node Scope service
Getting Mastership Status from Card Manager
Slot 27 HW arbitration result = Slave
Execute Remote console utlity from peer RP. This RP does not does not have MASTERSHIP

sysadmin-vm:0_RP0# attach location 0/RP1
Mon Aug 9 18:20:12.380 UTC+01:00
Last login: Mon Aug 9 16:52:52 2021 from 192.0.108.1
[sysadmin-vm:0_RP1:~]$ /opt/cisco/calvados/sbin/rconsole -l 0/4
Connecting to Card Manager Node Scope service
Getting Mastership Status from Card Manager
Slot 28 HW arbitration result = Master
Writing to console register
=> Address 0xfc set to 0x80050000
Connecting .. Escape char is ^x [control+x]

host:0_LC4 login:

Which is master/slave active/standby is a matter of timing and not something I suggest you worry about.