CSR1000v HA problem

[mhmservice]

Hi all

I have configured HA with CSR1000v as per the following guide (using the v2 instructions with the python scrips in the guest shell)

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-azure/b_csr1000config-azure_chapter_01001.html#id_79840

The problem I have is that the failover does not work correclty

If I reload the primary router, I get the following file created on the secondary router, so it is recognising the failure, but doesn't appear to actually be doign anything with the info:

[guestshell@guestshell events]$ more event.2019-06-03\ 15\:43\:00.261285

Event type is peerFail
index   100
routeTableName  FlexVPN
route   10.0.0.0/8
nextHop         10.192.226.12
resourceGroup   <redacted>
mode    secondary
subscriptionId  <redacted>
cloud   azure

Is there a way to troubleshoot this? There is nothing in the way of any useful logging I can find

I have checked the permissions of the VMs on the route table object in azure and it seems fine

Thanks in advance

[rms-aawebb · Accepted solution]

I've had this problem after our CSRs had to be moved to a different Availability Zone. I had to rebuild the CSRs, moved the configuration over 1 for 1, but the HA piece never seemed to kick off.

Everything looked like it was running (azure-ha, waagent and auth-token), but it looks like files and permissions were messed up in the guestshell:

Broken Guestshell (under the /home/guestshell/.local/lib/python2.7/site-packages/csr_azure_ha/client_api folder):

Spoiler

[guestshell@guestshell client_api]$ ls -lrt
total 54
-rwxrwxr-x 1 guestshell users 1228 Nov 17 2018 show_node.py
-rwxrwxr-x 1 guestshell users 1737 Nov 17 2018 set_params.py
-rw-rw-r-- 1 guestshell users 222 Nov 17 2018 revert_nodes.sh
-rwxrwxr-x 1 guestshell users 3028 Nov 17 2018 node_event.py
-rwxrwxr-x 1 guestshell users 8573 Nov 17 2018 ha_api.py
-rwxrwxr-x 1 guestshell users 1978 Nov 17 2018 delete_node.py
-rwxrwxr-x 1 guestshell users 2754 Nov 17 2018 create_node.py
-rwxrwxr-x 1 guestshell users 4009 Nov 17 2018 clear_params.py
-rw-rw-r-- 1 guestshell users 108 Nov 17 2018 __init__.py
-rw-rw-r-- 1 guestshell users 1560 Jun 21 18:55 show_node.pyc
-rw-rw-r-- 1 guestshell users 1868 Jun 21 18:55 set_params.pyc
-rw-rw-r-- 1 guestshell users 2999 Jun 21 18:55 node_event.pyc
-rw-rw-r-- 1 guestshell users 7184 Jun 21 18:55 ha_api.pyc
-rw-rw-r-- 1 guestshell users 2231 Jun 21 18:55 delete_node.pyc
-rw-rw-r-- 1 guestshell users 2457 Jun 21 18:55 create_node.pyc
-rw-rw-r-- 1 guestshell users 3821 Jun 21 18:55 clear_params.pyc
-rw-rw-r-- 1 guestshell users 317 Jun 21 18:55 __init__.pyc

Working Guestshell (same folder):

Spoiler

[guestshell@guestshell client_api]$ ls -lrt
total 61
-rwxrwxr-x 1 guestshell guestshell 1228 Jun 25 22:00 show_node.py
-rwxrwxr-x 1 guestshell guestshell 42 Jun 25 22:00 show_node
-rwxrwxr-x 1 guestshell guestshell 1737 Jun 25 22:00 set_params.py
-rwxrwxr-x 1 guestshell guestshell 42 Jun 25 22:00 set_params
-rw-rw-r-- 1 guestshell guestshell 222 Jun 25 22:00 revert_nodes.sh
-rwxrwxr-x 1 guestshell guestshell 194 Jun 25 22:00 py_caller.py
-rwxrwxr-x 1 guestshell guestshell 3028 Jun 25 22:00 node_event.py
-rwxrwxr-x 1 guestshell guestshell 42 Jun 25 22:00 node_event
-rwxrwxr-x 1 guestshell guestshell 8573 Jun 25 22:00 ha_api.py
-rwxrwxr-x 1 guestshell guestshell 1978 Jun 25 22:00 delete_node.py
-rwxrwxr-x 1 guestshell guestshell 42 Jun 25 22:00 delete_node
-rwxrwxr-x 1 guestshell guestshell 2438 Jun 25 22:00 create_node.py
-rwxrwxr-x 1 guestshell guestshell 42 Jun 25 22:00 create_node
-rwxrwxr-x 1 guestshell guestshell 4009 Jun 25 22:00 clear_params.py
-rwxrwxr-x 1 guestshell guestshell 42 Jun 25 22:00 clear_params
-rw-rw-r-- 1 guestshell guestshell 108 Jun 25 22:00 __init__.py
-rw-rw-r-- 1 guestshell guestshell 1560 Jun 26 14:19 show_node.pyc
-rw-rw-r-- 1 guestshell guestshell 1868 Jun 26 14:19 set_params.pyc
-rw-rw-r-- 1 guestshell guestshell 442 Jun 26 14:19 py_caller.pyc
-rw-rw-r-- 1 guestshell guestshell 2999 Jun 26 14:19 node_event.pyc
-rw-rw-r-- 1 guestshell guestshell 7184 Jun 26 14:19 ha_api.pyc
-rw-rw-r-- 1 guestshell guestshell 2231 Jun 26 14:19 delete_node.pyc
-rw-rw-r-- 1 guestshell guestshell 2235 Jun 26 14:19 create_node.pyc
-rw-rw-r-- 1 guestshell guestshell 3821 Jun 26 14:19 clear_params.pyc
-rw-rw-r-- 1 guestshell guestshell 317 Jun 26 14:19 __init__.pyc

Notice the additional files and the permissions are all set to the guestshell user.

I found that destroying the guestshell and rebuilding sometimes resolves this. I had to destroy the guestshell a couple of times for one CSR.

Steps:

guestshell destroy
guestshell enable

guestshell
    pip install csr_azure_guestshell~=1.1 --user
    pip install csr_azure_ha~=1.0 --user

I would then create a node and do the verify command, you should see a lot of additional information (API URLs, etc).

I wish this side was documented better.

HTH. 

-Aaron

[mhmservice]

I checked the directory you suggested above and I had a lower than expected number of files too (53 I think in my case)

I destroyed the guestshell and rebuilt .. configured the HA again, and now its working? Very strange...

I also upgraded to CSR1000v version 16.09.03 in the mean time (when I had this issue I was using 16.09.02) so maybe that was also part of the solution to the guest shell problems?

Whatever the cause, it works now! :)

Many thanks for your help