Hi Adham, Thank you for your comment, you can follow this document to upgrade your APIC CIMC, ensure to do each apic one by one, and wait till the cluster is fully-fit before starting your next apic. There should be no impact with running an older CIMC version, though it is best practise to move to the recommended CIMC versions as soon as possible, as you mentioned 3.0(3f) is the recommended CIMC versions for the APIC 3.2 software. (https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/recommended-release/b_Recommended_Cisco_ACI_Releases.html)
... View more
Just wanted to say, awesome document! Thank you for sharing this!! I also wanted to mention for those who get stuck, when creating the payload text file, make sure there isn't a new line created after the text, this will result in a 'bad signature' error when sending the api rest call to apic. The below command shows a new line was created for this file, ($).
[root@localhost ~]# cat -e payload.txt
To ensure no new line is created you can run the below echo -n command and redirect to a file.
[root@localhost ~]# echo -n "GET/api/class/fvTenant.json?rsp-subtree=children" >payload.txt
[root@localhost ~]# cat -e payload.txt
... View more
Thank you for your comment, a decommission of the Apic is not mandatory during CIMC upgrade, though it is recommended, decommissioning the Apic from the cluster is a more graceful method to shutdown the Apic, and ensures data replication is synced correctly across the remaining Apics. Just to make clear this procedure is for upgrading the APIC CIMC not to upgrade the APIC OS, just reach out or open a TAC case if you have any further queries in regards to this.
... View more
This document will provide screenshots to outline the steps to setup TACACS+ configuration to ACI and also the configuration required on Cisco ACS server.
Please find the official Cisco guide for configuring TACACS+ Authentication to ACI: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/2-x/Security_config/b_Cisco_APIC_Security_Guide/b_Cisco_APIC_Security_Guide_chapter_0100.html
*Note only specific web browsers and version are supported on Cisco ACS, check the release notes of your ACS versions to find the supported browsers. Using unsupported browsers can lead to configuration loss in ACS.
First step is to configure the APIC and fabric node devices on the ACS. Navigate to Network Resource > Network Devices Groups > Network Devices and AAA Clients
Specify the client name, and create a device type.
Add the Cisco APIC and fabric node Ip Addresses (out-band or INB), ensure to select IP range to add multiple node addresses.
Select TACACS+ under the authentication options, the shared secret will be used for the Cis co APIC provider Keys.
2. Next step is to configure users and Identity groups. Navigate to Users and Identity Stores > Identity Groups.
The identity group will group different types of users together, in the example below I have created an ADMIN Identity-group to group all users who will have admin privileges.
3. You can now configure the users, (these can be internal or external uses such as from Active Directory) navigate to Users and Identity Stores > Internal Identity Stores > Users. Create an admin user, map this user to the identity group, and configure a password.
4. Create the policy elements, this is where we configure the Cisco AV pair, to specify the APIC required RBAC roles and privileges for different users
The AV pair are configured in the following format:
shell:domains = domainA/writeRole1|writeRole2|writeRole3/readRole1|readRole2,
domainB/writeRole1|writeRole2|writeRole3/readRole1|readRole2(16003) * Unix ID
Below are some examples of the av-pair strings for some different types of users:
For users with admin access to entire fabric:
shell:domains = all/admin/(16002)
For users with read-only access to fabric:
shell:domains = all//read-all (16003)
*note readonly users do not have access to leaf’s or spines, admin write privilege is required.
For a user with admin access to tenants under security domain mydomain, and read only access to the rest of the tenants.
shell:domains = mydomain/admin/,all//read-all(16004)
* As best practice, Cisco recommend that you assign a unique UNIX user id in the range 16000-23999.If the UNIX user ID is not specified, ID 23999 is applied by the APIC system. There is a known bug if unique UNIX ids are not used.
Navigate to Policy elements > Device Administration > Shell Profiles and create a new shell profile. Name the profile, and navigate to the ‘Custom attributes’ tab, here you can add the av-pair string.
5. Last step is to configure the access policies, this is where you tie it all together, mapping the user to the shell profile to the AAA clients. Navigate to Access Policies > Default Device Admin > Authorization.
Create a new Rule, mapping the identity group, the device type and shell profile.
Configuring APIC for TACACS+ ACCESS
First create the TACACS+ Provider, navigate to Admin > AAA > TACACS+ Management > TACACS+ Providers.
Here you will need to specify the ACS hostname (ensure DNS is setup in your fabric if using hostname) or IP, the port, (default is 49), the key (this should match the ‘shared secret’ configured on ACS) and also the management endpoint (Inb or OOB)
*Note if the APIC has INB management configured, choosing the OOB management EPG for the TACACS+ provider does not take effect as INB is the default preferred by APIC. In versions 2.1(1x) and higher there is an option to toggle between INB and OOB so to make OOB as the default management connectivity. Find this under Fabric > Fabric Policies > Global Policies > APIC Connectivity Preferences.
2. Next create the TACACS+ Provider Group and map to TACACS+ provider created in step 1. Navigate to Admin > AAA > TACACS+ Management > TACACS+ Provider Groups.
3. Create the Login Domain and map to TACACS+ Provider group. Navigate to Admin > AAA > AAA Authentication > Login Domains.
*You will notice a ‘fallback’ login domain is already created by default, this is the local domain to allow local authentication in case you are locked out of your fabric if the default authentication settings are changed.
Changing the Default Authentication to your fabric.
If you would like TACACS+ to be the default authentication method when accessing the fabric you can make this change under the AAA authentication tab.
If TACACS+ is not configured as the default authentication method, (local is the default) you will need to explicitly select the login domain when logging in via GUI, or use the below format to access nodes using TACACS+ logins via cli.
Login format: ssh apic#domain\\username@ip
Recovering with Local Fallback User
In the event that connectivity is lost to the TACACS+ server and you are locked out of your fabric you can use the local admin fallback user. Below is the format to do this:
From the GUI, use apic:fallback\\admin.
From Cli, use ssh apic#fallback\\admin
Always ensure under Default Authentication that ‘Fallback Check’ is set to false, otherwise you will not be able to recover with local login credentials.
A couple of check to make sure your configuration is correct:
1. First check to make sure you can ping the TACACS+ Server from both Apic and Leaf nodes. This can rule out any network issues.
2. Check the configuration has been correctly pushed to the Leafs and APICs:
Leaf101# show tacacs-server groups
total number of groups:1
following TACACS+ server groups are configured:
server: 10.66.80.98 on port 49
deadtime is 0
Leaf101# show aaa authentication
3. Check the nginx logs and search for the TACACS+ provider IP to confirm reachability, these logs can sometimes give you a clue to why TACACS connectivity is not working.
leaf101# grep 10.66.80.98 /var/log/dme/log/nginx.log | less 34703||17-12-11 10:40:33.239+08:00||aaa||DBG4||||Received response from 10.66.80.98 - notifying callback handler (IPv4)||../dme/svc/extXMLApi/src/gen/ifc/app/./ping/lib_ifc_ping.cc||757 34703||17-12-11 10:40:33.239+08:00||aaa||DBG4||||Received update on status of 10.66.80.98 (DN uni/userext/tacacsext/tacacsplusprovider-10.66.80.98) - status is \ ALIVE||../dme/svc/extXMLApi/src/gen/ifc/app/./pam/PamWorker.cc||1448
... View more
Activity : Upgrade
Product (Cisco): ACI, APIC
APIC Servers are running on UCS C Series Servers, Cisco UCS 220 M4 (second generation appliances APIC-SERVER-M2 and APIC-SERVER-L2) or Cisco UCS 220 M3 (first generation appliance APIC-SERVER-M1 and APIC-SERVER-L1), with a minor difference that servers manufactured with an image secured with Trusted Platform Module (TPM), certificates, and an APIC product ID (PID).
Corresponding UCS Platform
Cluster of three Cisco APIC first generation controllers with medium size CPU, hard drive, and memory configurations for up to 1000 edge ports.
Cluster of three Cisco APIC second generation controllers with medium size CPU, hard drive, and memory configurations for up to 1000 edge ports.
Cluster of three Cisco APIC first generation controllers with large size CPU, hard drive, and memory configurations for more than 1000 edge ports.
Cluster of three Cisco APIC second generation controllers with large size CPU, hard drive, and memory configurations for more than 1000 edge ports.
Upgrading the software version of the ACI Fabric, might set new requirements on the CIMC version running on your Fabric, therefore, it is always advised to check the release notes of the APIC software version to know the list of supported CIMC software versions for a specific APIC release.
1. Check APIC Release notes, and confirm to which CIMC software image you need to upgrade. Link to APIC Release notes.
2. Obtain Software image from Cisco.com
3. Confirm that the MD5 Checksum of the image, matches the one published on Cisco.com
4. Time needed for the process of upgrading a CIMC version varies based on the speed of the link between the Local Machine and the UCS-C chassis, and source/target software image and other internal component versions.
5. Upgrading the CIMC version does not the production network as APICs are not in the Data Path of the traffic.
6. Please understand that changing CIMC version might also require changes to the Internet Browser, and Java Software version to run the vKVM.
APIC CIMC Upgrade Procedure
Upgrade the APIC CIMC using the Cisco Host Upgrade Utility.
Please find the official Cisco Document: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/sw/lomug/2-0-x/3_0/b_huu_3_0_1/b_huu_2_0_13_chapter_011.html
1. First step is to determine the APIC model:
APIC models S1/M1/L1 = C220 M3
APIC models S2/M2/L2 = C220 M4
You can check this in the CIMC GUI by verifying the PID displayed under Server --> Summary
2. Next you will need to download the appropriate HUU .iso image from https://software.cisco.com/download, this is located under C220 downloads.
Downloads Home - Servers - Unified Computing - UCS C-Series Rack-Mount Standalone Server Software - UCS C220 M3 Rack Server Software -Unified Computing System (UCS) Server Firmware - 3.0(4j)
*ensure to check the recommended CIMC version and software release guide to download the recommended CIMC firmware version for your current ACI software release.
* Please go to the recommended CIMC APIC firmware instead of the latest for that UCS
3. Launch the KVM console from CIMC GUI. (*If you are having problems opening the KVM console, this is generally an issue with your JAVA version. Please read the Release Notes for your CIMC version to learn the different workarounds available.)
4. Open KVM console and click Activate Virtual Devices, accept the session
5. Map the downloaded ISO on your PC, by selecting the 'Map CD/DVD 'and browse the HUU iso .
6. Reboot the server by using the macros tab and select Ctrl-Alt-Del
Note: If Ctrl-Alt-Del does not work you can use Power >> Power cycle System ( Cold reboot )
7. In the next step, we will need to press F6 to enter the boot menu so we can select the mapped DVD to boot from. You can create a user defined macro to do this if you are using a Remote Desktop application, otherwise you can just press F6.
8. The boot menu will prompt for a password, the default is 'password'.
9. Select KVM mapped DVD, as shown below.
10. Be patient. You will need to wait around 10-15 minutes while the ISO is extracted by the HUU, then another 15-20 minutes to copy the firmware and other tools
11. Accept the terms and conditions
12. It is recommended to update all the firmware for all components using the 'Update all' option.
* A pop up will be shown to enable Cisco IMC Secure Boot, select no, otherwise refer to the Introduction to Cisco IMC Secure Boot section in the Cisco UCS C-Series Servers Integrated Management Controller GUI Configuration Guide, Release 4.0
13. Wait till all the HUU updates PASS for each component, press exit, this will reboot the server.
When the server reboots you will be pushed out of the CIMC GUI, you will need to log back into the CIMC, and verify the upgrade has completed successfully. To do this you can verify via the GUI or boot up the CIMC host Upgrade Utility and select "Last Update Verify"to ensure all components passed the upgrade successfully.
... View more