01-13-2016 08:23 AM - edited 03-01-2019 06:04 AM
Introduction
This article will help you understand and educate the customer on how to clear their "expired licenses" (license-graceperiod-expired) from their UCS-M.
If a customer just purchased a license and needs a step by step guide on how to download them from the Product License Registration and install them on the UCS-M please reference the following link;
https://supportforums.cisco.com/document/12748516/installing-and-downloading-ucs-licenses
Problem Description
Customers often encounter errors in their UCS-M that notify them their licenses are expired even though all ports on their Fabric Interconnect are licensed. This causes most customers to worry if their environment will be effected by this critical error. However, this has no impact on the network and is just a cosmetic warning message.
These errors are usually related to the following bug ID's;
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCui19338/?reffering_site=dumpcr
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCus10255/?reffering_site=dumpcr
Problem
In this scenario the customer sees the following Critical error in their environment:
When you SSH into the Fabric interconnect you can verify it with the follow commands (shown in the picture below):
Scope License
Show Usage
This shows that Ethernet Ports on Fabric Interconnect A are the ones being expired. Now you can do even a deeper dive and figure out which ports on Fabric Interconnect A is the one currently down by running the following commands (Also shown in the picture below):
Scope eth-server
show interface
Based on the output above it tells you which port on a specific Fabric Interconnect are having the Expired License Issue Port-channel 1025 on port 15 is the one showing as expired, even though you can verify traffic is being pushed out of it.
Afterwards you want to run these following commands. What this shows is all the ports using a license that are ethernet uplinks and fibre channel uplinks. Each command has their respective output shown below.
scope eth-uplink
show interface
eku-ucs-A /eth-uplink # show interface
Interface:
Fabric ID Slot Port Admin State Oper State State Reason Lic State Grace Period
---------- ----- ----- ------------ ------------- ----------------------------------- -------------------- ------------
A 1 1 Enabled Up License Ok 0
B 1 1 Enabled Up License Ok 0
This shows the ethernet uplink ports on Fabric A and B, and both ports are configured as enabled, while their License status is set as Ok.
Now to troubleshoot the Fibre Channel ports on the Fabric Interconnects.
scope fc-uplink
show interface
eku-ucs-A /fc-uplink # show interface
Interface:
Fabric ID Slot Port Admin State Fill Pattern Oper State State Reason Lic State Grace Prd
---------- ----- ----- ------------ ------------- ------------- ----------------------------------- -------------------- ---------
A 1 41 Enabled Arbff Sfp Not Present
A 1 42 Enabled Arbff Sfp Not Present
A 1 43 Enabled Arbff Sfp Not Present
A 1 44 Enabled Arbff Sfp Not Present
A 1 45 Enabled Arbff Sfp Not Present
A 1 46 Enabled Arbff Sfp Not Present
A 1 47 Enabled Arbff Up License Graceperiod 3376800
A 1 48 Enabled Arbff Sfp Not Present
B 1 41 Enabled Arbff Sfp Not Present
B 1 42 Enabled Arbff Sfp Not Present
B 1 43 Enabled Arbff Sfp Not Present
B 1 44 Enabled Arbff Sfp Not Present
B 1 45 Enabled Arbff Sfp Not Present
B 1 46 Enabled Arbff Sfp Not Present
B 1 47 Enabled Arbff Up License Graceperiod 3376800
B 1 48 Enabled Arbff Sfp Not Present
This shows that two ports are running into the License Graceperiod Error, one on each Fabric Interconnect. This is due to the fact that the customer has "X" amounts of licenses yet he is going over that amount. This is because every time a port is configured and enabled it will use up a license even if its not being used to transfer data (you can notice this by the SFP Not Preesnt under the operational state). If your customer is running into this issue I would advise them to unconfigure those ports to disabled since they are not being used this will reduce the amount of licenses being used.
Solution
To solve the following issue you must complete the following steps.
1) Open an SSH/Putty session to your Primary Fabric Interconnect
2) UCS-B# show cluster state (Make sure this shows PRIMARY OK & HA READY)
3) UCS-B# connect local-mgmt
4) UCS-B (local-mgmt) # cluster lead a (This is if the current cluster lead is B so you would want to change it to A, this forces a failover). (This might kick you out from the putty & GUI session, it’s an expected behavior. Just wait couple of minutes and re login)
5) UCS-A# show cluster state (Make sure PRIMARY OK & HA READY, check the new primary & subordinate)
6) UCS-B# scope license
7) UCS-A /license # show usage (It should show license OK)
You will not need any downtime. It changes the management role and refreshes the grace period counter. It takes only 10 minutes. However, if this is a production environment and everything is running smoothly I would recommend doing this during a maintenance window since this expired license should not affect production.
Even though the bug ID states that you can bounce the ports with the issue (shut / no shut) along. I have found that the fail-over has been more effective and has worked for the majority of the time.
If this doesn't work and the customer is still seeing the error and he is in a maintenance window another known work around would be restart the 'licenseAG' service this can be done with a pmon stop / pmon start command (Careful this is a very DISRUPTIVE COMMAND!).
However, their have been issues where even a failover and a pmon stop / start would not clear these critical errors and then the only way to fix this cosmetic issue would be to upgrade the UCS firmware version to 2.2(4b) or higher, this bug is cleared in this version.
The bug links given on the doc are internal, please follow the external links:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCui19338/?reffering_site=dumpcr
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCus10255/?reffering_site=dumpcr
Can anyone post the commands that are in the NLA (no longer available) .png attachments?
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: