NSO high-availability token mismatch after node reboot

[Mircea Cristian]

Hi all, 

I am testing in the lab the high-availability feature of NSO, everything works fine until one of the nodes is rebooted.

After the nodes/node comes up I always get an error on the slave saying: Token mismatch and the salve node won't enter HA until I reconfigure the token on the master/slave nodes.

Configuration used on both nodes:

ncsadmin@ncs-labmaster# show running-config high-availability 
high-availability token <some-random-token-used>
high-availability ha-node ncs-labmaster
address 172.16.200.1
nominal-role master
!
high-availability ha-node ncs-labslave
address 172.16.200.2
nominal-role slave
failover-master true
!
high-availability settings enable-failover true
high-availability settings start-up assume-nominal-role true
high-availability settings start-up join-ha true
high-availability settings reconnect-interval 10
high-availability settings reconnect-attempts 6

NSO and tailf-hcc pacakge version:

ncsadmin@ncs-labmaster# show ncs-state | include ver
ncs-state version 5.7.3

ncsadmin@ncs-labmaster# show packages package tailf-hcc 
packages package tailf-hcc
package-version 5.0.3
description "Package for Tail-f HA Cluster Control Interface"
ncs-min-version [ 5.4 ]
oper-status up

It doesn't matter witch node I restart, once it comes back and tries to join the cluster, I get the token mismatch error.

For the lab env I'm using Centos7.

Is there some configuration I'm missing or anyway to debug this further? (I always commit the token before enabling HA)

Thanks,

[cohult · Accepted solution]

See https://developer.cisco.com/docs/nso/guides/#!high-availability/nso-built-in-ha under "HA Member Configuration".

In an NSO system install setup, not only the shared token needs to match between the HA group nodes, the configuration for encrypted-strings, default stored in /etc/ncs/ncs.crypto_keys, need to match between the nodes in the HA group too.

The token configured on the secondary node is overwritten with the encrypted token of type aes-256-cfb-128-encrypted-string from the primary node when the secondary node connects to the primary. If there is a mismatch between the encrypted-string configuration on the nodes, NSO will not decrypt the HA token to match the token presented. As a result, the primary node denies the secondary node access the next time the HA connection needs to reestablish with a "Token mismatch, secondary is not allowed" error.