Howdy out there in automation land! And welcome to the first blog of 2020... I know its been too long without one, but we have had so many things going on and lots I want to write about. And of course, make videos about them... so I just had to pick one thing and off we go. Today we are going to show a great example of collaboration across our Customer Experience (CX) organization! But first... a movie poster to go with this? Well this is about teamwork and one of my favorite movies and favorite teams was...     Love the Blues Brothers! And Blue Brothers 2000 is a highly underrated movie... not for the acting... but the music! Anyways, back on topic. So what we want to talk about today is collaborating on a problem, designing a solution, and deploying it using Action Orchestrator.   The Problem...   So this week we ran into an issue where our production CCS/AO system went fully down... not because of anything we did, but because of something in Kubernetes that we should have been monitoring for. The certificates for the client side API calls expired. Thusly making our micro-services unable to talk to each other or the Kubernetes cluster itself... well that was not good! And yes, we did open a bug, CSCvt21378, on this so you guys can track it once it becomes customer facing! Anyways, so the whole cluster goes down... wow! That is not good.   So I started to dig through tech articles, Kubernetes settings, and working with other Cisco folks to start designing a "fix" in the knowledge that other customers with private cloud based CloudCenter Suite(CCS) systems would see the same issue. It was time to go into proactive mode to start looking for how we could detect this and get ahead of it! A little bit of downtime to implement a workaround/fix was OK... but a whole day or more is just unacceptable!   The Solution   After a solid day of working on it I designed a workaround/solution to fix the above issue. It will be coming out public as a knowledge article in short time or you can get the information through TAC... however that was not *GOOD* enough. We in CX can do better! We can innovate... we can AUTOMATE! In talking with a fellow CX Engineer, Jason Davis, he had an idea to throw together a quick bash script to check on the certificate expiration dates for each cert. What a novel and great idea!!! Here is a quick plug for his twitter account - https://twitter.com/snmpguy , he's a great follow! Anyways he designed this script: for crt in /etc/kubernetes/pki/*.crt; do printf '%s: %s\n' \ "$(date --date="$(openssl x509 -enddate -noout -in "$crt"|cut -d= -f 2)" --iso-8601)" \ "$crt" done | sort Nothing overly fancy, but highly useful! So he had something in place to start looking at certificate dates... but we needed someone to make this reusable and deploy-able to our customer base....   Solving it with Action Orchestrator   Knowing that Jason is a huge AO person as well... we decided to collaborate and team up on this issue. How can we take a small script, package it for usage, and pass it out to the masses? Easy... we can write a workflow to handle it all!! And then publish it... so how do we start? We start with determining what the date was and how far away that was from the expiration date... we run the script to determine the dates of the certificates as well. We can use the Unix/Linux target to connect into one of our masters and run this bash script... like this:     We then needed to use regular expressions to filter out the data we needed and make it usable. In the video I will reference a great testing site that you can use, it is here: https://regex101.com/. Make sure when you do your regular expressions or look at the code that you use GoLang! (As that is what AO is using) Then we have to look at the dates and do some similar logic to determine whether we are in a "ok" time period, we need to "warn" the administrators, or the certificate is flat out expired! (OH NO!) We can use a standard condition branch in this case and compare the dates in them to determine the outcome. To be fancy... we are going to create an HTML based email to send to the administrator and run this workflow daily to let them know what is going on...       And lastly we are going to send that nice HTML email to the administrator(like we said above)... it would look something like this: Pretty slick huh? The best part of all this... the work that Jason and I put in, is sharable with you! So if you have CCS/AO, which you probably do if you are interested in my blog, you can go and import the workflow and script we created and use it in your environment to monitor your Kubernetes certificates. You can find it here: https://github.com/cisco-cx-workflows/cx-ao-shared-workflows/tree/master/CCSCheckKubernetesExpiration__definition_workflow_01E01VIRWZDE24mWlsHrqCGB9xUix0f9ZxG     So awesome... now you have a solution that went from a major problem to a simple script to find the certificate expiration dates to a sweet team effort to create a proactive monitoring workflow to keep your CCS system up and running! How neat is that?? You know what is just as neat? The video where I give you all the insight and talk you through the workflow... so now... ONTO THE VIDEO!   Play recording Recording password: WmkexiV2   Just a quick ~17 minute video explanation for you. Reminder that the content above in github is opensource in nature and is *NOT* supported by TAC. It was written by some great folks in CX but is open for you to use, take, manipulate, etc. So please... enjoy it!   Again, special thanks to Jason Davis for the collaboration here and a wonderful CX Solution!   Standard End-O-Blog Disclaimer:   Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)   AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, development, devops, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com   Thanks and Happy Automating!!!   --Shaun Roberts shaurobe@cisco.com   ... View more

  Howdy out there in automation land! So it is that time of year again. Snow for many, rapid weather changes for some, happy holidays for all. I hope the end of this year finds you in good spirits, good health, and ready for better automation. But first... a good movie poster for this...     Just a pure classic.... and... well because its that time of year and I'm in the giving mood, here's one more to close out the year...   And yes folks... it is a Christmas movie :) Ok.... enough with the posters... onto the reason behind this blog. So I really want to start pushing the community as a whole to start sharing your content! Let's make workflows great for everyone and share. We in Cisco CX have started to do this and I wanted to share our repo with you as well as share a couple of the workflows I wrote to help you make your life easier in Action Orchestrator (AO).   First off... the CX Shared and OpenSource repo is at... https://github.com/cisco-cx-workflows/cx-ao-shared-workflows   There you will find lots of examples, sample workflows, and workflows to help grow your automation portfolio. *PLEASE NOTE* none of these are officially supported by Cisco TAC but you can raise issues on them in the git and we can fix them as a community!   Now this is different than the *official* Cisco Workflow Git at https://github.com/cisco/ActionOrchestratorContent That repo is where you will find officially TAC supported workflows that are produced by Cisco for your usage. This is also where you find the atomic adapters.   You are welcome to post to whatever repo you personally own with your workflows but I would love to see more posting to the CX or a similar opensource repo so we can really start to share the content! This is how we grow the AO user base, how we grow AO's capabilities and how we just make AO better.   So let's start with the couple of workflows I'm showing and telling in this blog...   1st one - Install Compiled Adapter This workflow will take a compiled adapter in a .tar.gz bundle and then explode it, install the schema, and setup the adapter for you, assuming you have a compiled adapter already in a container format. Now if you have not gotten into using compiled adapters yet or do not really even know what that is... maybe best to just watch the video and understand the explanation... but in brief you will need...   The workflow above The adapter in .tar.gz bundle as per the video The compiled container stored somewhere Access to a master node, Access to your AO API Watch the Video :) This will allow you to install compiled adapters like the Webex Teams adapter I wrote. If you have interest in using the Webex Teams adapter, please reach out to me directly or post in the comments below. We used it live at Cisco Live US this year, and its ok to distribute in a controlled fashion and for controlled usage :)   Video here: Play recording Password: 3xXtPbqk   2nd workflow - Install Atomic Adapter   This one is more readily usable by the masses in my opinion. You install the workflow, configure your AO API endpoint, and then you can install the Atomic Adapters released by Cisco in an "easier" format. Instead of having to install 20 workflows individually you can just install the "kubernetes" adapter. It works as follows...   Install the above workflow Configure your AO API user and Target (use the video for help) Run the workflow and type in your adapter. Could be "aci" or "kubernetes" or "appd", etc. Enjoy!   Video on this workflow: Play recording Pass: ChCVUB3n   If you run into any particular issues/etc with these, please post them back to the repos or leave a note here! Once again, I hope you have a wonderful holiday season and come back in 2020 ready to automate as much as you with AO! Tis the season for giving and I hope giving of workflows can help start you getting into the holiday spirit!   Standard End-O-Blog Disclaimer:   Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)   AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, development, devops, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com   Thanks and Happy Automating!!!   --Shaun Roberts shaurobe@cisco.com ... View more

  Howdy out there in automation land! So I know it has been a while since my last blog but wow... the places and things we are doing... as I sit here this morning (early in the morning mind you)... I'm getting mentally prepared to teach a 4th Action Orchestrator boot camp to fellow Cisco engineers and partners. This is the 4th one in the past year alone.... to total about 230+ engineers having been trained on this wonderful product. It is amazing to see the excitement and drive behind this product and to be a part of the wave that it is creating.... get on the bandwagon now... do not let it pass you by!!! So let's get onto the meat of this blog and some other stuff, but first... gotta have a movie poster right? How about...     An oldie but a goodie... and on DVD no less! (Anyone under the age of 30 is probably asking what is that right now but I digress). Anyways, onto the what this blog is about. I believe you should practice what you preach. How can I teach all these engineers and you my awesome readers about AO if I'm not using it myself to make my life easier? Well that is what I have for you today. In doing these boot camps, I have students do labs and build Kubernetes from scratch, then install AO, then do workflow authoring, and finally do some adapter development. (which is super cool stuff and I plan to do some blogs on it, but that is another topic for another day). In doing that I am building small 3 noded clusters (1 master, 2 workers) and deploying those to an internal lab network I have. I also need to update the nodes with the latest information/latest versions, and all that good "update" jazz. (in CentOS the basic yum -y update command). I started to think on how I could make this easier for me because I was having to tear up and rebuild over 60 VMs for each boot camp. While I don't have them all the time... it did get to be a multi-day time sink for me and was hard to do other things(multi-task) while building them.   Enter AO. So in looking around, our labs have APIs to deploy and remove VMs (like most virtualization would). What we use is really not relevant to the discussion although you will see it in the video... we have cloud forms over VMWare. So my goal for standing up a VM was to... Make sure the VM did not exist already Read information about the VM I wanted from a SmartSheet via API. (SmartSheet is like Excel/Project in the cloud if you d not use it) Take that information and use it to deploy a template VM I built prior Wait for that VM to be ready Login to that VM via SSH and do some basic networking and VM configuration Change the VM IP address and restart its network. Change the IP address in my AO target dynamically and reconnect Run the yum -y update command and then mark the VM as ready to go!   Whew! That's a lot. But it is something that is highly repeatable and makes for great automation. So I sat down and in about a day's time I was able to do it all! A complete run to build a VM takes <20 minutes and while that is going on I can be doing a bunch of other work... I was able to build up a complete cluster, practice my labs, and tear in all down in a matter of 60 minutes. The time savings has been amazing! Previously it would have taken me about 1 week to build up the full lab and that is 1 week of me actually *DOING* the work. This time it took maybe 2 days of running the Workflow in AO. Now I ran it one at a time instead of all at once, just to make sure it did what I wanted. (first time was exciting!) But in the future I'll set it to run all 20 PODs and will probably take less than a day to do all that work. The tear down... even faster. It tears down and removes all the VMs in less than a day, including proper cleanup.   Wow... so that is a lot. This blog is truly about showing you, my readers, one of the exciting and highly useful things I've been able to do with AO. It is to give you ideas and get you even more excited about automation in AO. If you have not gotten into it... contact Cisco, contact us, come see us at Cisco Live EMEA or Cisco Live US this next year. Let's get you on the AO train, because it has pulled out and has a full head of steam behind it.   Now... I wanted to show you in live action this above automation... and how I built it and walk you through it. So....   ONTO THE VIDEO! Play recording Password: PyFu3i3H   Note: It's about ~30 minutes so sit back with a nice cup of coffee and enjoy.   Standard End-O-Blog Disclaimer:   Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)   AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, development, devops, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com   Thanks and Happy Automating!!!   --Shaun Roberts shaurobe@cisco.com   ... View more

  Howdy out there in automation land!!!! Again... two in one day... wow :) So onwards we press. If you have not read Part 1, please go back and do that as it might not make sense. In this part of the Less is More series we are going to install CloudCenter Suite (CCS) and Action Orchestrator(AO) to our 4 noded Kubernetes cluster we setup last time.I think we'll use this movie poster... because after you do this, you are truly going to be "The One" :)       At a high level we are going to follow the install guide found here. Remember single commands are in BOLD and longer commands are blocked. Without further delay... let's get onto it!   Prerequisites First we need Kubernetes version 1.11.3 or later... no worries! We should have installed 1.15 or later. We need to install Cert Manager v0.5.2 via kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.5/contrib/manifests/cert-manager/with-rbac.yaml We must define a default storage class, you are welcome to use VMWare or Openstack or whatever cloud as a storage class but in this case I will suggest using local disk or what we call "Host-Path Provisioner". You can find out about all storage classes here. So in this case, we will assume we have a 2nd drive mounted on /mnt/disk2 in each node. To use the host-path provisioner and really follow this end to end, do these sub-tasks, if not, skip to step #4 If you run the command kubectl get sc you will probably see nothing! But we need storage for databases for CCS. Use the attached yaml (or see below) and run kubectl create -f host-path-provisioner.yaml. If you look at the POD spec, you will see volume mounts for /mnt/disk2. You should change that to whatever your volume mount point is when you apply the yaml. apiVersion: v1 kind: ServiceAccount metadata: name: hostpath-provisioner namespace: kube-system --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: hostpath-provisioner namespace: kube-system rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: hostpath-provisioner namespace: kube-system subjects: - kind: ServiceAccount name: hostpath-provisioner namespace: kube-system roleRef: kind: ClusterRole name: hostpath-provisioner apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: hostpath-provisioner namespace: kube-system rules: - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: hostpath-provisioner namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: hostpath-provisioner subjects: - kind: ServiceAccount name: hostpath-provisioner --- # -- Create a pod in the kube-system namespace to run the host path provisioner # -- replace "/mnt/disks" with whatever your host path is apiVersion: v1 kind: Pod metadata: namespace: kube-system name: hostpath-provisioner spec: serviceAccountName: hostpath-provisioner containers: - name: hostpath-provisioner image: mazdermind/hostpath-provisioner:latest imagePullPolicy: "IfNotPresent" env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: PV_DIR value: /mnt/disk2 volumeMounts: - name: pv-volume mountPath: /mnt/disk2 volumes: - name: pv-volume hostPath: path: /mnt/disk2 --- # -- Create the standard storage class for running on-node hostpath storage apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: namespace: kube-system name: manual annotations: storageclass.beta.kubernetes.io/is-default-class: "true" storageclass.kubernetes.io/is-default-class: "true" labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: EnsureExists provisioner: hostpath There is one as well for windows, it is included in the zip below. You can now run kubectl get sc again and you should see the provisioner setup. You can also see a pod running for it. We now need to create the priority classes for CCS, we do this via cat <<EOF | kubectl apply -f - apiVersion: scheduling.k8s.io/v1beta1 kind: PriorityClass metadata: name: suite-high value: 1000000 globalDefault: false description: "High priority" --- apiVersion: scheduling.k8s.io/v1beta1 kind: PriorityClass metadata: name: suite-medium value: 10000 globalDefault: false description: "Medium priority" --- apiVersion: scheduling.k8s.io/v1beta1 kind: PriorityClass metadata: name: suite-low value: 100 globalDefault: false description: "Low priority" EOF You can then run kubectl get pc to check them.   Install CloudCenter Suite and Action Orchestrator You will now need to load the Installer VM which can be downloaded from Cisco.com. You would probably either use the OVA or qCOW2, so download what you need. Load the install VM up either into vCenter or whatever hypervisor you want. If you don't have a hypervisor you can use a "local" hypervisor like Virtual Box or VMWare Fusion. I used vCenter, the only requirement is that the installer can reach the IP address of your master. Once your installer VM launches, it should grab an IP address and then you can go to https://<IP-ADDRESS> to load the installer in the web Select GET STARTED under the Existing Kubernetes Cluster header Slide the Load Balancer as a Service option to NO Go back to your master VM and go to your Kube home directory via cd $HOME/.kube Print out your admin configuration via cat config and then copy and paste that into a text file and save as adminConfig.json Go back to the installer web UI and click Select File and then select your json file from step 7. Then click INSTALL Most likely you will get the system to install to the "default" namespace. If it does not, then replace "default" in commands with the proper namespace If you are using host-path provisioner, you might run into an issue with a pod and a database not being on the same host. If you run into this follow some of these sub-steps to "fix" it. I ran into it on one install and one install I did not, so hopefully you do not, but if you do... Check if postgres is running or not via kubectl -n default get posts -o wide | grep "postgres" If postgres is running, just skip the rest and go back to the main steps, if it has errors, continue. Please note: If you have other pods failing due to pods and storage not being on the same node, you can use this for any deployment or stateful set. To fix postgres issue... we must determine the nodes they are assigned to. The above command will show you what node postgres is trying to run on. Now we need to find where the storage is. Run the command kubectl get pv | grep "postgres" to find the PV ID. Then describe the PV via kubectl describe pv <PV ID>. This will show you the assigned node in the annotations section. You can see in the above it is assigned to "ao-so-rtp-09", so we need to force the pod to be on the same node. To do this we need to run kubectl -n default edit statefulset common-framework-suite-postgresql . Look for the priorityClass line and above it, insert a line like nodeName: <node> Then save the stateful set and it should bounce the pod and start running. If you run into this, make sure you watch the video as I detail what I did, etc. Once the bottom bar of the installer completes, it will prompt you to go to your registration URL and will send you to https://<master-ip>:<node port of ingress> Register your root tenant admin and login the first time! Congrats :) To then install AO, go to the Dashboard once you login and click install next to AO and pick the version to install. Once it is completed you can go into AO.   Wow!!! Now you have your own "non-production"/smaller cluster to work with AO on CCS directly!!! If you run into any issues, please feel free to post questions/comments/etc into the comments section of the blog.I highly suggest using the VODs as a guide and work alongside them... now... ONTO THE VIDEO!!!   Play recording (38 min 54 sec) Recording password: HppRgUP2       Standard End-O-Blog Disclaimer:   Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and now those learning AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)   AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com   Thanks and Happy Automating!!!   --Shaun Roberts shaurobe@cisco.com ... View more

  Howdy out there in Automation land!!! Today... I have the start of a long set of two blogs for my readers. We are going to do something exciting and really useful... but purely about system setup and design... no real "automation" today. But first... we have to have our movie poster. Since the goal of these is 30 minutes or less... I think this will do:     So what are we going to do today? We are going to install a 4 node (or VM) cluster of Kubernetes and then install CloudCenter Suite(CCS) and Action Orchestrator(AO) on it! (the CCS install will be on the Part 2 of this 2 part series). The main goal here is to allow you guys to create smaller and more agile clusters that you can use to run development, stage, pre-production style environments, or in the event that you do not feel you need all the compute power that the self-driven CCS cluster install gives you. For this... single line commands will be in BOLD. Anything multi-line will be broken out separately.   A reminder... when you install CCS and have it do the Kubernetes setup, it will make 3x masters (in HA), 1x internal load balancer, and 5-8+ workers, so you are looking at 12-14 VMs normally with lots of compute space and need. While this makes a ton of sense for production deployments, we want something smaller for non-production... and that is what I'm going to help you with today! Pleas note these steps are accurate as of this blog post... Kubernetes/Docker/Linux change often so I might have to update or just point you to other places. You can always find the Kuberentes documenation for setup here.   First, like any good recipe, we need the ingredients, so what you will need is... 4x Virtual Machines (on any cloud) These VMs should have... 4x-8x vCPU, 8-16GB Ram, a base 50GB hard dive and a 100GB secondary storage hard drive. 1 IP address per VM is fine. It is easier to have IPs that are NATed to the internet versus behind a proxy, but we can address both scenarios. I highly suggest 8x vCPU and 16GB RAM per node, but it will work with the lower power if you must. These VMs should have CentOS 7.5 or later installed. They can be a pretty basic install for a compute node. If you need guidance in adding an additional hard drive, follow the post here. You could do this on windows VMs or other linux VMs, but this blog will not take you through that. If you need windows Kubernetes guidance, look here. You could also look into pre-constructed packages, but again, this part of the blog is install of Kubernetes from scratch. You will want to configure each /etc/hosts file with the IP addresses and hostnames of the 4x nodes. These VMs should be updated to the lastest codebase via yum -y update You will need root or sudo access... prefer root, it's just easier! I would suggest you take the above "setup" and copy/clone it. Do it once and make it a template so you can spawn these nodes as you need them. (you might even want to do a template below right before kubeadm init) Docker Install You will need to install the docker prerequisites via yum install -y yum-utils \ device-mapper-persistent-data \ lvm2 Add the docker-ce repo, we will install docker-ce 18.06 to be our container runtime yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo Install Docker and enable it via  yum update && yum install docker-ce-18.06.2.ce systemctl enable docker && systemctl start docker You can check the docker version via docker --version Setup docker to use systemd as a cgroupdriver via cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ] } EOF Then restart docker via systemctl daemon-reload and then systemctl restart docker If your nodes are behind a proxy, you will most likely have to follow these sub-steps. If you do not have a proxy, skip to the Kubernetes install. You can test this by doing curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 and see if you get back anything. Proxy Sub Steps - you can also find this guidance here. Create a directory for docker via mkdir -p /etc/systemd/system/docker.service.d Add PROXY configuration to your /etc/profile, something like this: HTTPS_PROXY="http://myproxy.cisco.com:80" HTTP_PROXY="http://myproxy.cisco.com:80" NO_PROXY="127.0.0.1,localhost,<List of 4x VM IPs>" export HTTPS_PROXY HTTP_PROXY NO_PROXY Inside of your docker.service.d folder, create two files, http-proxy.conf and https-proxy.conf Inside of them, you must add the proxy configuration like this (for HTTPS). Do the same for HTTP [Service] Environment="HTTPS_PROXY=http://myproxy.cisco.com:80" "NO_PROXY=127.0.0.1,localhost,<YOUR SYSTEMS IP ADDRESSES>" After making these changes, restart and flush the daemon via systemctl daemon-reload and systemctl restart docker. You can then verify the changes via systemctl show --property=Environment docker You can also try the curl again at the top and it should work. Kubernetes Install Add Kubernetes to your yum repo via cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kube* EOF Turn off setenforce via setenforce 0 for this session. Turn if off permantenly then via sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config Find the Swap Space via free -h and blkid. You will see something like this: /dev/mapper/centos-root: UUID="390ed2f2-7111-465d-9bfc-0b2d5d9285ac" TYPE="xfs" /dev/sda2: UUID="isHFES-V66X-ZSE7-R8T1-fh0b-HztV-vFo2US" TYPE="LVM2_member" /dev/sdb1: UUID="af19aa1f-759d-4612-9d83-ebf96fd83324" TYPE="ext4" /dev/sda1: UUID="dcdd2d1e-41b7-48dd-920b-d784b3f60ba8" TYPE="xfs" /dev/mapper/centos-swap: UUID="41322535-274e-498a-9b76-7b6a6dc5ca1e" TYPE="swap" /dev/mapper/centos-home: UUID="707eae72-2fd4-4112-acc0-170b1783661e" TYPE="xfs" You will need to disable the swap space via below. You can find an article on it here. swapoff <swap area> #most likely /dev/mapper/centos-swap #so most likely swapoff /dev/mapper/centos-swap swapoff -a #Permantly disable via sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab Install Kuberadm, Kubectl, Kubelet via yum install -y kubelet kubeadm kubectl cri-tools --disableexcludes=kubernetes Kubeadm is the cluster initialization module, Kubectl is the CLI tool, and Kubelet is the agent/service Enable and set Kubernetes to auto-start via systemctl enable kubelet && systemctl start kubelet We need to setup the call-iptables for CentOS routing for Kubernetes via cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system You have a choice to disable and stop the firewall or figure out *ALL* the ports and punch a hole for each one... I highly recommend disable/stop via systemctl stop firewalld and then systemctl disable firewalld. If you really want to punch holes in your firewall, below is a start... but you might come across other ports you need. Much easier to do step 9 instead... but if you want to keep the firewall up, start with this: firewall-cmd --zone=public --add-port=6443/tcp --permanent firewall-cmd --zone=public --add-port=10250/tcp --permanent firewall-cmd --zone=public --add-port=2379/tcp --permanent firewall-cmd --zone=public --add-port=2380/tcp --permanent firewall-cmd --zone=public --add-port=8080/tcp --permanent firewall-cmd --zone=public --add-port=80/tcp --permanent firewall-cmd --zone=public --add-port=443/tcp --permanent firewall-cmd --zone=public --add-port=10255/tcp --permanent firewall-cmd --reload This is not a bad place to do a template/snapshot. You could deploy clones from this step, but not after. Start your cluster via kubeadm init --apiserver-advertise-address=<MASTER-IP> --pod-network-cidr=192.168.1.0/16 You need to fill out the master IP address (the box you are running this on) and then set the pod CIDR since we will be using Calico for our network. You will see a lot of output... you want to grab parts after it says "Your Kubernetes control-plane...". We need the mkdir command and then kubeadm join command. The kubeadm join will be used on your worker nodes. Using the mkdir command, we run this to give us access to the admin config: export KUBECONFIG=/etc/kubernetes/admin.conf mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config Now you can run kubectl get nodes and see your single node. (It should be NotReady) I *highly* suggest using the kubectl auto-complete via kubectl completion bash and then kubectl completion bash >/etc/bash_completion.d/kubectl . This makes using kubectl a lot easier!!!! Now we have to install Calico, grab it via curl https://docs.projectcalico.org/v3.7/manifests/calico.yaml -O And then install it via kubectl apply -f calico.yaml You should see coredns pods coming up and calico nodes come up when you do kubectl -n kube-system get pods Setup Workers And Install Kubernetes Dashboard Now we need to stand up our workers and then an optional part, the Kubernetes dashboard You should setup all the workers with the same base as the above. Then proceed to do the docker install the same. Then proceed to do the Kubernetes Install and stop after step 10(firewall step) and proceed with step 3 below. You should have copied the join command from the above (when you init the master) and now you should run this on each worker... kubeadm join 10.122.138.195:6443 --token xv7uhj.9tzey70dka4v9gh7 \ --discovery-token-ca-cert-hash sha256:055f6fc799bbfff68ba93f84ea2c07389bfceb2226235df0e2868091f126f412 NOTE: Your IP, TOKEN, HASH will all be different than the above, it is just an example. You should see some more kube init text but it will be smaller. On your master, you can run kubectl get nodes to see your workers being added. Once you have done step 3 on each worker, proceed to step 5. Step 5 on is purely OPTIONAL. It is to install the Kubernetes dashboard which is a web based UI of Kubernetes. You do not have to have this for CCS, if you do not want this... you are done! (you can move onto the next blog-part 2) If you want the dashboard, continue on... Install Kubernetes Dashboard via kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml We need to edit the dashboard service to expose it to the outside world, so let's do that via kubectl -n kube-system edit service kubernetes-dashboard This will open VI up and you need to go find the part that says type: ClusterIP and change it to say type: NodePort Save it via hitting ESC, then wq! Now we need to find the port! Run kubectl -n kube-system get service kubernetes-dashboard. You will see a ports list and something like 443:XXXXX/TCP. That means your dashboard is running on https://<IP-OF-MASTER>:<XXXXX> But first, we have to configure a service user for us to access the dashboard... Kubernetes is very strict on RBAC, etc I have attached a yaml file to the blog that you can use, and it is below... but you need to apply it. So if you use the file just do kubectl create -f clusterRoleBindingForDashboard.yaml otherwise here is the yaml: apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard labels: k8s-app: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: kubernetes-dashboard Please note this setup is meant to be a little "open"... remember, dev/stage environment, not PRODUCTION! Lastly we need to bind the cluster role to the account so run this command: kubectl create clusterrolebinding varMyClusterRoleBinding \ --clusterrole=cluster-admin \ --serviceaccount=kube-system:default Then we need to get the token or password for the dashboard, we can do that via kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep kubernetes-dashboard | awk '{print $1}') Just copy out the "token" at the very end of the output of the above command Now we can go to our dashboard, which remember, is like https://<ip-of-master>:<xxxxx>, at the prompt paste in your token and sign in. I suggest saving that login... :) You now have a fully working Kubernetes cluster with dashboard! Congrats to you!!!!   Now you guys did not think I would leave this without a VOD right? I'm going to guide you through it!!! So ONTO THE VIDEO: This VOD covers docker+master install+calico: Play recording (20 min 23 sec) Recording password: kJE6Wbbc   This VOD covers worker install: Play recording (7 min 28 sec) Recording password: uBpejFT3       Standard End-O-Blog Disclaimer:   Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and now those learning AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)   AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com   Thanks and Happy Automating!!!   --Shaun Roberts shaurobe@cisco.com ... View more

Howdy out there in Automation land!! Hope everyone is having a great summer. We draw into the last true month of summer and we are going to take you further on your Action Orchestrator journey. Since we are on our last "Back to the Basics", I think we will go with this movie poster...     Going a little old(er) school for you guys this time! In this blog we are going to do my most recent "advanced" workflow authoring lab. This will teach you looping, conditionals, and how to work with tables in AO. Also to go along with that, I would like to introduce the new *LOGO* for the blog and get your feedback! :)   Let me know what you folks think! Lots of pondering in that Thinker fella. Also in the video blog I will chat a bit on using git integration with your workflows. I know we already did that but a quick refresher. I will also talk, briefly, about how to get your "easier to use" API keys from CloudCenter Suite so you can use them with AO calls *outside* of AO. I know you can do them in AO via the API page, but this is from outside or if you want to call AO from itself..... *Twilight Zone Music*.   To get our API key we need to do the following... Login to our CloudCenter Suite Select Suite Admin Click your login name at the top right Select Generate API Key The User ID it gives you is your username. The API Key is your Password You can now use those in a Basic Authentication style header. So base 64 encode username:password and you are good to go! (Or just use anything like postman/etc that does it for you) Now... onto the lab!! (got you there I bet... Video is coming..)   Setup Steps Create a new workflow called Advanced Workflow Lab Add it to your category or any Create some variables! A string for output called id_list. A local integer called LVC. A couple of local strings for storage. An input integer called number_of_users. A local integer called TRUE, set it to 1. The local strings can just be local_storage or whatever you like to call them Writing an ADVANCED WORKFLOW! You will need to re-use the JSONPath holder HTTP endpoint we made in the Back to the Basics Part 2 blog. Don't have it? Go back to that blog and watch and add it!!! Now... back to this lab... Make a call to JSONPath Holder again accessing the /posts endpoint. It should be a GET. *You* get to fill in all the other stuff... watch the VOD here or do this first and attempt it yourself. This will help you and should not be hard if you watched the previous blog. Now add  new data type to your your workflow, click ADD VARIABLE under Variables, under Data Type click ADD NEW. Table Type will be selected as it is the only advanced data type currently supported. Give your table a DISPLAY NAME. This table is just for show, we won't use it, and you will see in the VOD I skipped it... however I want you to do it so you get the feel of creating a table! Go to your COLUMNS and add fields via ADD 1 MORE FIELD at the bottom. The FIELD NAME is the column name, the FIELD TITLE is what is on the UI. You can select the type for the column in FIELD TYPE and add optional parameters. Add a few fields and make some required and some not via the REQUIRED slider. Click SAVE to save your table. Now back to your workflow....Drag and drop a Read Table from JSON activity below your web call. Under Read Table, set the JSON PATH to $..table[*] and check the PERSIST TABLE checkbox. If you do not check this, the data from the table will not be available to you later in the workflow. In COLUMNS TO READ click ADD to start adding columns. They should be all type String and their names should be userId, id, title, and body. Their names must match the JSON. Under SOURCE JSON, we need to "wrap" the web request json in some tags to make it readable, so we will put the output of the web activity into the SOURCE JSON field, but then wrap it as below { "table": <reference to web call output> } What about a LOOP and the Results? Drag and a drop FOR EACH Loop from the Logic area in the toolbox under your Read Table from JSON activity Click the insert variable reference icon to load in the SOURCE ARRAY. Select the output of the Read Table from JSON Activity. You can *only* feed tables into this field. Drag and drop a CONDITION BLOCK inside the FOR EACH loop Set the DISPLAY NAME of the conditional to If ID LIST is empty Set the left CONDITION BRANCH to THEN and right one to ELSE (Remember we are doing that best practice of documentation! --- and all the developers gasp!) On the left CONDITION BRANCH set the CONDITION to one of your local variables(the storage one) being empty or blank. We can do that by setting it to Equals then a blank field. On the right CONDITION BRANCH set the CONDITION to your local TRUE variable Equals to 1. This is another way to do an ELSE statement. In the left or empty branch, drag and drop a SET VARIABLE activity. Set your local string variable to the current loop iterations id value. You do this via Activities->For Each Loop->Source Array->Items->id. Then click SAVE. On the right branch, drag and drop another SET VARIABLE activity, or duplicate the other one. Set your local string variable to itself, then a comma, and then the current loop iteration id value. This will build you a comma-delimited list of ids. This is how we do string concatenation in AO and will be *HUGE* for you going forward. How about looking at the table where ID=2, how do we do that? Drag and drop the Select from Table activity below your loop Under Select, set the INPUT TABLE to the output of your Read Table From JSONPath Then set the WHERE CLAUSE to id='2'. Persist this table as well and return all columns. Lastly, set an output variable. Set your output id_list to your local storage. Validate, Run your workflow and see the results! Check out individual iterations of the loop. Check out what is at ID=2 and check out your resultant id list!     Pretty cool huh? A quick and simple lab to teach you some of the advanced things or standard flow control in AO! Now time for you to build on it more!!! Feel free to try the above or watch the video and do it at the same time...   Now.... ONTO THE VIDEO!!!! :)   Link: Play recording Pass: sQJSapc2    Standard End-O-Blog Disclaimer:   Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and now those learning AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)   AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com   Thanks and Happy Automating!!!   --Shaun Roberts shaurobe@cisco.com ... View more

Howdy out there in automation land!!!! So.... To Atomic or Not To Atomic.. that is the question? Is it not? This post is going to break into the string of "back to the basics" as we have gotten a lot of interest in Action Orchestrator of late (which is stellar) and we have a lot of customers interested in building their *OWN* adapters. The question is... do you do atomics or compiled/custom adapters or what? And what is best for you? That is what I plan to investigate and give you guys information on for this blog. You have multiple options with AO... so I want you to make the best one for your automation needs!   For a movie poster.... let's see, that line at the top sounds like something I might have pulled from some famous play....     I think works pretty well right? This is the version I liked the best out of all the movie versions.   So let my preface the two "sides" of the fence I'm going to talk about. Atomic Adapters and Compiled or Custom Adapters. To start there is two places you can look for (in documentation) to see these constructs... Atomic Adapter Compiled Adapter   So what is the difference and what do you get out of one versus the other? Atomic Adapters Let me start with Atomic Adapter. An atomic adapter is made up of Atomic workflows. One of the key benefits of it is that you do not need *ANYTHING* else other than AO to make it work. You build workflows in AO, mark them as Atomic and you are done! How do you do that? 1. Open up your workflow 2. Go to the main workflow properties 3. Click the "Is atomic workflow" checkbox and give it a Group. What is a Group? That is how the atomic workflow will be organized in the UI. This will position it in your ACTIVITIES section as a drag and drop component. The group will be the heading it will be under. So where and when do these fit best? In my opinion, if AO can make the calls and already has all the functionality you need built in, then these make the most sense. They are also quickest to develop and deploy and require no extra coding knowledge/skill/etc. If you are building an atomic adapter you should make all your atomic workflows very "micro" workflow in nature. In other words, do just 1 operation, maybe wrap it in a little error handling if you like... and be done. Do *NOT* make them overly complex or do a lot. They should be a bunch of micro workflows or activities in nature only. A word of caution... make sure the workflow is tested and good to go. Once you check "atomic" and make it an atomic workflow, you cannot click the "RUN" button anymore to test it. You can always unmark it from being an atomic but in my experience I suggest fully vetting out the workflow, then checking the atomic box and moving onto the next thing. The drawbacks to Atomic Adapters is that they are bound to what AO currently can do and bound to the AO basic UI. If you feel you need something a little more advanced... then the next section is for you!   Compiled Adapters   Also called "Custom" adapters in the documentation. These are adapters you *BUILD FROM SCRATCH*! So it is an exciting feature but pretty advanced in nature. I might have a series on how to do this coming out on the blog later this year.. but I need to figure out how to make it most effective for my great readers! So what do I mean, build from scratch? Well at a high level you must do the following things in creating a customer adapter.... Get the adapter template code base (GO, Python, or Java) from Support (TAC). Cisco provides you with a basic foundational structure in one of those 3 languages so all you have to worry about is filling out the code for your main parts. We provide the dockerfile, yaml for Kubernetes, basic code, etc in these templates. Write Schema for your adapter as you have to "teach" AO how to understand your adapter. You must create them in the order of ACCOUNT KEYS, ADAPTER, TARGET, then ACTIVITIES and EVENTS. You can do this via the UI under Admin->Schema or you can do this directly via API call.  You can make the API call against the /schemas endpoint and the body would need to be both schemas(data and view, see below). Design activities for your users in the UI via the schema Write the appropriate code (GO/Python/Java) for your target, adapter, account keys, and activities. Compile your adapter as needed Convert your compiled adapter to a docker image Deploy your new docker image to a Pod via Kubernetes into your CCS/AO setup As par of your schema creation you have to understand that you would need to create both "DATA" and "VIEW" schema. Data is what is passed across the wire and view is what makes up the UI/UX. I am not going in too deep on this post because that is not what this is about... this post is about you making the right decision for your use-case on what path for you to take! So what is the value here.... the pros? The pros are pretty much anything is possible. You can bring in packages from other languages and design your activities pretty much in whatever you want. I find custom adapters are best when... You have a pre-existing SDK for some system/API that you can just wrap the adapter code around The system you want to plug into has some kind of authentication scheme or endpoint scheme that AO does not support yet. You want to do some advanced work with the UI/UX on your activities You want to create highly customized targets, events, etc So am I pushing you in one direction or the other? Of course not! I have wrote adapters in both!!! Cisco does as well. The adapters you have when you install AO are custom or compiled adapters. The adapters we also offer on our public git at https://github.com/cisco/ActionOrchestratorContent are atomic adapters. They include ones like ACI, UCSD, VMWare, Azure, etc. For those you can just plug into our git and download at will... not authentication required! For me personally, I have wrote a couple atomic adapters for our internal systems that just use basic API calls and what not.... on the custom side I have been working on the following adapters: Webex Teams, Intersight, MongoDB, and CSOne (our support case system). Both CSOne and Intersight are great examples of the SDK "reason" to use custom adapters. In Intersight there is already a great python SDK we publish...so why re-invent the wheel??? Word of advice, don't! :) Just wrap it in the adapter code and use it. Same thing with CSOne. I wrote a python API wrapper package for it and instead of keeping up multiple code bases, I just wrap it in adapter code and now just keep up the python package and I have less work to do but I can serve multiple user bases. The other side of the coin with custom adapters is that is it is a very advanced concept and does require a bit more work and knowledge of coding.   Overall, I think you should be fine to use either one. Pick what best fits your use-case. Use them both! Use AO a ton! But with the amount of questions and queries I've been seeing from customers I thought it was time to put this on paper... or blog in this case. If you have any questions on these, please post below or hit me up on my email or social media below!   As always I have a video on the same and dive into it a bit deeper with you.... I also (in the VOD) will give you a sneak peek look inside some of the great AO work that was done at Cisco Live US this year in San Diego... we did some major stuff, including pushing an ROI of just under 1.25 million during the show... using AO only in the span of just 6 days. Can you imagine doing the same for your IT department??? Why would you not want that? :) Do you want to know how? You have to watch the video :) If you have interest in AO or getting plugged in... please reach out to me and I will get you plugged into the right folks! My email and social contacts are always posted at the end of the blog.   As always... ONTO THE VIDEO!   Play recording (25 min 35 sec) Recording password: mP7dgTdh       Standard End-O-Blog Disclaimer:   Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and now those learning AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)   AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com   Thanks and Happy Automating!!!   --Shaun Roberts Twitter: https://twitter.com/LordAutomation LinkedIn: https://www.linkedin.com/in/srobertsmba/ Email: shaurobe@cisco.com ... View more

Howdy out there in automation land!!! I hope everyone is having a nice ramp down to the school year and is getting ready for summer. I know we are :) I hope plenty of you folks are heading to Cisco Live in San Diego in a few weeks too!!!! I plan to be there, so feel free to hit me up on email or look me up in the NOC. I would love to meet some of the readers and viewers and have automation chats. So what do I have for you this time? This time we are going to look at workflow creation... basic workflow creation.   Now what movie poster would this be you say? Well I like to think of new workflows as a "New Hope" for automation... so how about this?     This might be a bit of a replication of some of the *really* old CPO stuff, but Action Orchestrator is a *NEW* product. It is the NextGen, but is net new as well. So I want you guys to treat this as such. On top of that, know that (unless otherwise) most of my blog going forward will be centered around AO. So get excited!   I will give you guys some steps to follow here... but of course there is a video below as well :)   Now, for this, something you can do first is create the users and targets. In the recording I already had them and talked to creating them on the fly, but you might find it easier to create them first... for this "lab" we need a web target(http) and an email server.   Setup Steps Create a category under Admin->Categories and use your CEC ID as the category. Create a runtime user under Account Keys. Click NEW ACCOUNT KEY, select HTTP Basic Authentication and fill out the information. The actual information does not matter. You can use test/test if you like. Click SUBMIT. Create a target under Targets. Click NEW TARGET, select HTTP Endpoint and fill out the information. Set DEFAULT ACCOUNT KEYS to your user from #2. Select HTTPS protocol. Set the HOST/IPADDRESS to jsonplaceholder.typicode.com. Set the PORT to 443 and select DISABLE SERVER CERTIFICATE VALIDATION. Click SUBMIT. Create another target under Targets. Click NEW TARGET, select SMTP Target and fill out the information. Under DEFAULT ACCOUNT KEYS click the Add New option and use your CEC email/CEC Password. Then select that user. Set the SMTP SERVER to outbound.cisco.com and set the SMTP PORT to 25.   Let's Create a Basic Workflow   Create a new workflow under Workflows. Click NEW WORKFLOW. Go down the right side and select your category under the CATEGORY drop down. Under Variables click ADD VARIABLE. For the first variable Select Data Type as String. Display Name as path and SCOPE as Input. Set the value or default as users. Click SAVE. Click ADD VARIABLE again. Select Data Type as String. Display Name as output json and SCOPE as Output. Click SAVE. Continue down to Target. Select Execute On This Target and then select HTTP Endpoint under the TARGET TYPE drop down and then select your target under the TARGET drop down. Search for http in your toolbox and drag and drop the HTTP Request activity. This is the basic activity to make any API call. Click on the HTTP Request activity. Change the DISPLAY NAME to GET JSONPATH. Scroll down to RELATIVE URL. Click the puzzle-piece icon(or insert variable reference) and a new dialog should appear called Browse Variables. Click Workflow, then Input, then you should see your path variable. Click it and then click SAVE. You should now see a VARIABLE REFERENCE showing in your RELATIVE URL field. Set METHOD to GET. Scroll down and set the CONTENT TYPE to JSON. Search for the Set Variables activity. Drag and down it BELOW your GET JSONPATH activity. Change the DISPLAY NAME to Set Output. Click ADD. For VARIABLE TO UPDATE, click the insert variable reference icon, and browse to your output variable. Workflow->Output->output json. Click SAVE. For VARIABLE VALUE NEW, click the insert variable reference icon. Now browse to Activities, then your activity GET JSONPATH, then select Body. Click SAVE. Create a new variable of type Integer and call it Status Code. It should be an output scope. Add your new variable to the Set Output activity. Click ADD below your first entry and set your Status Code Variable to the value in Activities->GET JSONPATH->Status code. Validate the workflow and run it. See what your variables are set to.   Let's get it running in layered!!! Exit your current workflow and create a new one called "Master" or similar Scroll down and select Execute On This Target. Select HTTP ENDPOINT as the TARGET TYPE and then select your JSONPATH holder target in the TARGET select. Drag and drop your workflow to your canvas. It should look a little different. Add an input variable call name of type string. Find the JSONPath Query activity. Drag and drop it below your atomic workflow in the MASTER workflow. On the JSONPath Query activity, click insert variable reference and point it to the output of your previous activity. This should be Activites->Your Activity->output json. Click SAVE. Scroll down and click ADD below JSONPATH QUERIES. Leave the PROPERTY TYPE as String. Make the PROPERTY NAME be name and make the JSONPATH QUERY be $.[?(@.username==“<input name>)]. Select the checkbox for CONTINUE WORKFLOW EXECUTION ON FAILURE Go the middle icon in your toolbox, this is the Program Logic or Flow Control area. Find the Condition Block and drag and drop it below your JSONPath Query. Click the Condition Block in your canvas and change the Display Name to If found name. Click on the left most CONDITION BRANCH and you should see the conditional dialog. Under CONDITION, title it THEN, then select the insert variable reference icon for the first field and map it to Activity->JSONPath Query->Succeeded. This is a boolean value that tells us if the activity succeeded or not. Select Equals in the second field and then type true into the last field. Click on the other CONDITION BRANCH, and do the same as #5, but set the last field to false. Title this block ELSE. This will work because the JSONPath query activity will error if it does not find what you are looking for. Search for the EMAIL activity. Drag and drop it into the left CONDITION BRANCH blocks. On the email activity, go to Target and select Override Workflow Target. Select your SMTP server from above. Scroll down and input your email into the TO field. Input something into the SUBJECT line and something into the MESSAGE field. Something like We found them! On your email block, click the ... and select duplicate. Drag the duplicate over to the other CONDITION BRANCH. (Your else branch) Change the SUBJECT and MESSAGE to reflect not finding the person. Validate and run the workflow. Search for bill. Did it email you? Now run again and search for Bret.   This is a great lab that I adapted from some things I built before but I hope you were able to get through it!!! If you run into issues or have questions, please post below. As always, I have a VOD to go along with this of course :) .... now.... ONTO THE VIDEO!   Play recording Password: JiEUHzy2    Standard End-O-Blog Disclaimer:   Thanks as always to all my wonderful readers and those who continue to stick with and use CPO and now those learning AO! I have always wanted to find good questions, scenarios, stories, etc... if you have a question, please ask, if you want to see more, please ask... if you have topic ideas that you want me to blog on, Please ask! I am happy to cater to the readers and make this the best blog you will find :)   AUTOMATION BLOG DISCLAIMER: As always, this is a blog and my (Shaun Roberts) thoughts on CPO, AO, CCS, orchestration, and automation, my thoughts on best practices, and my experiences with the products and customers. The above views are in no way representative of Cisco or any of it's partners, etc. None of these views, etc are supported and this is not a place to find standard product support. If you need standard product support please do so via the current call in numbers on Cisco.com or email tac@cisco.com   Thanks and Happy Automating!!!   --Shaun Roberts shaurobe@cisco.com ... View more