Howdy out there in automation land! I hope everyone is doing well. A reminder... Cisco Live goes live 6/2 at 9am PDT... join us online for 2 great days at CiscoLive.com/us. I will update this blog once my session is released so you readers can check it out too! So our topic today is going to cover you "moving on up" to Action Orchestrator 5.2!!! It was just released on Friday(5/30) and I've already migrated a couple of mine and made these VODs for you.... Migrated you say? Yes... that's how it works! But before we get into that let's get our movie poster up.... hmmmm... "moving on UP"?
My kids love that movie! So here is the thing on you moving to 5.2... you have two options..
1. You: I've never installed or used AO yet
ME: Great! Start today!!! You just need to install CCS and install 5.2 directly. You are done. Don't have to read the rest
2: You: I've been running AO 5.X or 5.1.X and I am ready to upgrade!
ME: Great!!! This blog is for you. We will go through the migration that is documented here. You *cannot* upgrade directly *WITHOUT* the migration.... so please don't try :) Why a migration? Because you are moving from a single arango service to a clustered arango service.
So please note, this migration is officially documented, TAC supported... all those goodies. So you should be able to do it without worry! But... being a fairly technical thing, I thought there might be value in recording and sharing some experiences I had over the weekend of doing it. I've migrated 3x clusters to date!
So let's get started...
- Before you start doing anything... make sure you are on Suite Admin 5.2... if not, upgrade. I'll wait for you :) Also make sure you have CLI based access to one of your master nodes in your cluster. If not, get with an admin as you will need access.
- First you will need Arango DB client tools. You can get them here. While you can get them on your local box either for windows or something else... I would suggest you do your arango work directly on a master node. It will just be faster. Most likely you will be doing this via CentOS or Ubuntu however so those are detailed on the documentation. So if you wanted to install the client for CentOs it would be like...
cd /etc/yum.repos.d/ curl -OL https://download.arangodb.com/arangodb36/RPM/arangodb.repo yum -y install arangodb3-client
For ubuntu, follow the doc, but for CentOS you can just use the above as well. - You need to open up the Arango Service port (most likely)
So to do that we do...
kubectl -n <namespace> get svc | grep "arango" #this gets the name kubectl -n <namespace> edit svc pers-arangodb #this will open the editor to change it
Once you open it, look for "ClusterIP" and change it to "NodePort". Then hit ESC, and then exit and save. (colon and then wq!)
This is what it should look like after the fact...
apiVersion: v1 kind: Service metadata: creationTimestamp: "2019-11-04T12:10:17Z" labels: app: pers-arangodb chart: pers-arangodb-3.4.5-1 heritage: Tiller release: action-orchestrator name: pers-arangodb namespace: ccp resourceVersion: "26215995" selfLink: /api/v1/namespaces/ccp/services/pers-arangodb uid: 8e5e3d0e-a6cc-4dd2-add6-fd0bdba2beaf spec: clusterIP: 10.106.1.231 externalTrafficPolicy: Cluster ports: - name: pers-arangodb-svc nodePort: 30308 port: 8529 protocol: TCP targetPort: pers-arangodb selector: app: pers-arangodb release: action-orchestrator sessionAffinity: None type: NodePort status: loadBalancer: {} - You will need to run the kubectl -n <namespace> get svc | grep "arango" again. Note the new PORT now showing. It should be in the 30000+ range. You need that port so write it down!
- After saving you will need to get your secret key via
kubectl -n <namespace> get secret action-orchestrator-jwt-secret -o jsonpath="{.data.jwtSecret}" - Then you need to decode it via
echo <jwtSecret from step 5>|base64 --decode
- Now you will need to install nodejs and npm.
- *IF* you are behind a proxy, make sure you have set your HTTPS_PROXY host environment variable like...
export HTTPS_PROXY="http://myproxy.example.com:80"
- If you are on a DEBIAN based machine (CentOS For example)...
#run these steps... curl -sL https://rpm.nodesource.com/setup_10.x | sudo bash - sudo yum install nodejs #verify via node --version
- If you are on a UBUNTU based machine...
#run these steps... sudo apt install nodejs npm sudo apt install build-essential #verify via node --version
- *IF* you are still behind that proxy, you will need to set the npm proxy... otherwise skip this. But if you need to set it then...
#run this sudo npm config set https-proxy http://myproxy.example.com:80
- Now install jwtgen via
npm install -g jwtgen
- *IF* you are behind a proxy, make sure you have set your HTTPS_PROXY host environment variable like...
- Take the decoded value you got from step 6 and then run this to get a bearer token...
jwtgen -a HS256 -s <DECODED KEY from step 6> -c server_id=setup -c iss=arangodb
- Now you will run an API call via CURL to update the root password for arango like this. You can make "somepassword" be whatever you want... just remember it :) The user will be "root"
curl -X put http://<master IP>:<exposed NodePort>/_api/user/root -H 'Authorization: Bearer <TOKEN GENERATED FROM STEP 8>' -d '{ "passwd": "somepassword" }' - Now on your master... you want to backup arango. So you will run this command...
arangodump --server.endpoint tcp://<master IP or hostname>:<exposed port> --server.username root --server.password <password you set in step 8> --all-databases true --threads 4 --output-directory arango_backup
I used "arango_backup" as the folder but you can call it whatever you want. It will then store the backup of your arango DB into that folder.
Up to here... this is covered in my first video...
Play recording
Recording password: Qcqcmc2w - Now you need to go and COMPLETELY uninstall AO. Login to your CCS cluster in the root tenant. Go to Suite Admin. Click the drop down next to AO and click UNINSTALL.
- To verify things are gone you can run some of the following commands...
kubectl get pods -n <namespace> | grep action-orchestrator kubectl get services -n <namespace> kubectl get pvc -n <namespace> kubectl get pv -n <namespace>
- As an update to the doc/video I recorded... it is *OK* to have the secrets stick around... you do not have to get rid of them. You do need to make sure all PVC/PVs and all Pods/deployments are gone.
- After those are gone... wait about 5 minutes. Then go back into Suite admin and click the install button on AO and select 5.2. It will install!
This is covered in this video...
Play recording
Recording password: Bjn2WmfJ - After AO is back up and running... meaning the last pod is running. (That is BE-bootstrap pod), we need to scale down the services to prepare for migration... so run these commands...
kubectl scale deployment -n <namespace> action-orchestrator-be-lds --replicas=0 kubectl scale deployment -n <namespace> action-orchestrator-be-console --replicas=0 kubectl scale deployment -n <namespace> action-orchestrator-be-bootstrap --replicas=0 kubectl scale deployment -n <namespace> action-orchestrator-be-orchestrator --replicas=0 kubectl scale deployment -n <namespace> action-orchestrator-be-rbac --replicas=0 kubectl scale deployment -n <namespace> action-orchestrator-be-event --replicas=0 kubectl scale deploy -n <namespace> action-orchestrator-be-schedule --replicas=0 kubectl scale deploy -n <namespace> action-orchestrator-be-importexport --replicas=0
#You can then verify they are down via...
kubectl -n <namespace> get deployments | grep "action-orchestrator-be" - You are going to repeat what we did in step 3 above to re-open the arango port. Change it from ClusterIP to NodePort again. The only difference is the name is now
action-orchestrator-pers-arangodb. - Run this command to get the root password for your arango cluster...
kubectl -n <namespace> get secret action-orchestrator-pers-arangodb-root-password -o jsonpath="{.data.password}" | base64 --decode - Execute the restore of your arango DB via...
arangorestore --server.endpoint ssl://<master-ip>:<port> --server.username root --server.password <password from step 16> --all-databases true --overwrite true --create-database true --replication-factor 3 --threads 4 --input-directory <backup folder from earlier>
- It should succeed... but if you see any errors, run it again. It is *HIGHLY* suggested you do the backup and restore to one of your masters since it will be fastest and have less chance to have network errors.
- You are now *ONLY* going to upscale the LDS... via
kubectl scale deployment -n <namespace> action-orchestrator-be-lds --replicas=2
- You now can go to https://<master-ip>:<exposed arango port> and you should see arango UI. You can login with "root" and the password you got in step 16.
- You then can go to queries and run the following query to check for successful ones...
FOR i in lhtenants FILTER i.enabled==true AND i.deleted==false AND i.status.state == "migration-successful" return i
You can see most of this restore on this video....
Play recording
Recording password: wH9CcMHA - Once you have successful migrations... you need to turn AO 5.2 back up... so such by starting the BE services back up...
kubectl scale deployment -n <namespace> action-orchestrator-be-console --replicas=2 kubectl scale deployment -n <namespace> action-orchestrator-be-orchestrator --replicas=2 kubectl scale deployment -n <namespace> action-orchestrator-be-rbac --replicas=2 kubectl scale deployment -n <namespace> action-orchestrator-be-event --replicas=2 kubectl scale deploy -n <namespace> action-orchestrator-be-schedule --replicas=2 kubectl scale deploy -n <namespace> action-orchestrator-be-importexport --replicas=2 #You can then verify they are down via... kubectl -n <namespace> get deployments | grep "action-orchestrator-be"
NOTE: DO NOT SCALE UP BOOTSTRAP YET! - Now you can scale bootstrap up as a 1 of... like kubectl -n <namespace> scale deployment action-orchestrator-be-bootstrap --replicas=1
- After these are all up and running... you should have a successfully migrated 5.X->5.2 AO setup! Enjoy!!
- You should also make sure if you have tenants and other users that you re-share AO or re-give them access to AO. Because you uninstalled and reinstalled its like it is a new application all over again.
- You can set the arango port back to ClusterIP if you do not wish to have it exposed anymore. Up to you.
All of this rescaling is seen on this video...
Play recording
Recording password: uRRxXmX3
Since I've integrated the videos into the steps above... no "ONTO THE VIDEO" :( ... but its ok... look forward to an upcoming blog where we will discuss new stuff/features in 5.2!
Standard End-O-Blog Disclaimer:
Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)
AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, development, devops, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com
Thanks and Happy Automating!!!
--Shaun Roberts
