topic Re: NSO Geo HA - HA connection lost in NSO Developer Hub Discussions

NSO Geo HA - HA connection lost

bchai — Fri, 01 Mar 2019 11:54:09 GMT

Upon activation of Geo HA, the connection established, NSO-1 (Master), NSO-2 (slave), NSO CDB synced, no operation on NSO and no route change, in approx 30 sec, NSO-2 reported that ha connection lost and it it’s switching to master. While this is happening, we verified the connection between the 2 NSO-1 and NSO-2 is reachable.

not sure if we're missing any config ?

Error message as below:

admin@SNGKC# ha commands status

status sngkc-nfv-nso-2[slave] connected sngpc-nfv-nso-1[master]

admin@SNGKC# e*** ALARM node-failure: HA connection lost. 'sngkc-nfv-nso-2' transitioning to HA MASTER role. When the problem has been fixed, role-override the old MASTER to SLAVE to prevent config loss, then role-revert all nodes. This will clear the alarm.

Version:

NSO: 4.2.3.1

Tailf-hcc package: 4.0.5

Quagga-bgp package: 3.4.0

NSO Config:

admin@SNGPC# show running-config ha

ha token tokenxyz

ha local-user admin

ha bgp anycast-prefix 10.234.3.255/32

ha bgp anycast-path-min 2

ha member sngkc-nfv-nso-2

address 101.234.3.242

default-ha-role slave

relay-name sngpc-nfv-nso-1

quagga-device srdm-quagga2

ha member sngpc-nfv-nso-1

address 101.234.3.238

default-ha-role master

cluster-manager true

quagga-device srdm-quagga

admin@SNGPC#

Re: NSO Geo HA - HA connection lost

Dan.Sullivan — Thu, 29 Jun 2017 15:47:32 GMT

Hello,

So just to be clear initially HA connects and then 30 seconds later it lose connectivity? Usually its some sort of port issue firewall settings, etc. but this doesn't seem to be that if the connectivity is established and then drops.

-Dan

Re: NSO Geo HA - HA connection lost

lmanor — Thu, 29 Jun 2017 22:12:45 GMT

More information is needed here, can you provide snippets of the devel.log and ncs-java-vm.log at the time of the event.

-Larry

Re: NSO Geo HA - HA connection lost

bchai — Fri, 30 Jun 2017 01:55:21 GMT

Upon checking, the ha config "ha bgp anycast-prefix" configured on both NSO-1 and NSO-2 could wrong, possibly causing this issue, although that doesn't explain why the short lived HA connection was able to establish and CDB was synced to the slave in the first place.

will rectify the "ha bgp anycast-prefix" on both NSO-1 and NSO-2 and retry.

Further info from previous test:

HA status:

admin@SNGKC# ha commands status

status sngkc-nfv-nso-2[slave] connected sngpc-nfv-nso-1[master]

from devel.log:

NSO-1:

<INFO> 27-Jun-2017::03:10:37.257 sngpc-nfv-nso-1 ncs[15158]: ncs HA_INFO_SLAVE_ARRIVED: <<"sngkc-nfv-nso-2">> {101,234,3,242}

<INFO> 27-Jun-2017::03:11:02.290 sngpc-nfv-nso-1 ncs[15158]: ncs HA_INFO_SLAVE_DIED: <<"sngkc-nfv-nso-2">> {101,234,3,242}

<INFO> 27-Jun-2017::03:12:12.391 sngpc-nfv-nso-1 ncs[15158]: ncs HA_INFO_SLAVE_ARRIVED: <<"sngkc-nfv-nso-2">> {101,234,3,242}

<INFO> 27-Jun-2017::03:12:14.616 sngpc-nfv-nso-1 ncs[15158]: ncs HA_INFO_SLAVE_DIED: <<"sngkc-nfv-nso-2">> {101,234,3,242}

<INFO> 27-Jun-2017::03:13:19.365 sngpc-nfv-nso-1 ncs[15158]: ncs HA_INFO_SLAVE_ARRIVED: <<"sngkc-nfv-nso-2">> {101,234,3,242}

<INFO> 27-Jun-2017::03:13:50.293 sngpc-nfv-nso-1 ncs[15158]: ncs HA_INFO_SLAVE_DIED: <<"sngkc-nfv-nso-2">> {101,234,3,242}

NSO-2:

<INFO> 27-Jun-2017::03:11:07.689 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_IS_NONE

<INFO> 27-Jun-2017::03:11:12.200 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_SLAVE_INITIALIZED

<INFO> 27-Jun-2017::03:11:36.960 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_IS_MASTER

<INFO> 27-Jun-2017::03:12:41.576 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_IS_NONE

<INFO> 27-Jun-2017::03:12:47.304 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_SLAVE_INITIALIZED

<INFO> 27-Jun-2017::03:12:49.285 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_IS_NONE

<INFO> 27-Jun-2017::03:13:48.987 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_IS_NONE

<INFO> 27-Jun-2017::03:13:54.254 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_SLAVE_INITIALIZED

<INFO> 27-Jun-2017::03:14:24.964 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_IS_MASTER

<INFO> 27-Jun-2017::03:14:50.059 sngkc-nfv-nso-2 ncs[6399]: ncs HA_INFO_IS_NONE