cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Troubleshooting and Clearing UCS Faults

11971
Views
0
Helpful
0
Comments

     

     

    Introduction

    This document shows a few common UCS faults and method to clear them using CLI. Since the faults can not be manually deleted through the UCS Manager GUI or CLI, this document shows CLI steps to clear these faults. Each fault represents a failure in the Cisco UCS instance or an alarm threshold that has been raised. During the lifecycle of a fault, it can change from one state or severity to another. Fault includes information about the operational state of the affected object at the time the fault was raised.

    Fault Severity

    Following table explains fault severity in UCS

    SeverityDescription
    CriticalA service-affecting condition that requires immediate corrective action. For example, this severity could indicate that the managed object is out of service and its capability must be restored
    WarningA potential or impending service-affecting fault that currently has no significant effects in the system. Action should be taken to further diagnose
    MajorA service-affecting condition that requires urgent corrective action
    MinorA non-service-affecting fault condition that requires corrective action to prevent a more serious fault from occurring
    ConditionAn informational message about a condition, possibly independently insignificant
    InfoA basic notification or informational message
    ClearedA notification that the condition that caused the fault has beenresolved, and the fault has been cleared

     

    UCS Faults

    If port channels and pools are configured in UCS, but not used, they will show up as either a Major or a Minor fault.

    UCS1-FA-A# show fault
    Severity  Code     Last Transition Time     ID       Description
    
    --------- -------- ------------------------ -------- -----------
    Major     F0282    2012-02-08T13:59:23.094   1576322 lan port-channel 2 on fabric interconnect B oper state: failed, reason: No operational members
    Major     F0282    2012-02-08T13:58:11.915   1576231 lan port-channel 2 on fabric interconnect A oper state: failed, reason: No operational members
    Minor     F0465    2012-04-12T16:52:38.892    193850 IP pool iscsi-initiator-pool is empty
    Minor     F0476    2012-05-09T20:21:15.985   1668798 FC pool node-wwn-assignment node-default is empty

     

    Steps required to clear the fault:

    Fault #1 - Port Channel with No Operational Members
    
    1. Log onto the primary fabric interconnect through UCS Manager CLI with administrative privileges.
    2. Disable the port channel/s with "No operational members".

     

    Fault #2 - IP Pool iscsi-initiator-pool is empty
    
    1. As a UCS user with administrative privileges, create a bogus block of IP address in the iscsi-initiator-pool, such as: 1.1.1.1 to 1.1.1.1.

     

    Fault #3 - FC pool node-wwn-assignment node-default is empty
    
    1. As a UCS user with administrative privileges, create a bogus WWN entry in the node-default pool, such as: 50:00:25:B5:11:11:11:11.

     

    Note: The changes mentioned above are disruptive, please take adequate precautions.

    Clear "Port Channel with No Operational Member Fault" on Fabric Interconnect B

    UCS1-FA-A# scope eth-uplink

    UCS1-FA-A /eth-uplink # scope fabric b

    UCS1-FA-A /eth-uplink/fabric # show port-channel

    Port Channel:

        Port Channel Id Name         Admin State Oper State       State Reason

        --------------- ------------ ----------- ---------------- ------------

        1               port_ch_1    Enabled     Up

        2               port_ch_2    Enabled    Failed           No operational members

    UCS1-FA-A /eth-uplink/fabric # enter port-channel 2

    UCS1-FA-A /eth-uplink/fabric/port-channel # disable

    UCS1-FA-A /eth-uplink/fabric/port-channel* # commit-buffer

    UCS1-FA-A /eth-uplink/fabric/port-channel # exit

    UCS1-FA-A /eth-uplink/fabric # show port-channel

    Port Channel:

        Port Channel Id Name         Admin State Oper State       State Reason

        --------------- ------------ ----------- ---------------- ------------

        1               port_ch_1    Enabled     Up

        2               port_ch_2    Disabled   Failed           No operational members

    UCS1-FA-A /eth-uplink/fabric # exit

    UCS1-FA-A /eth-uplink # exit

    UCS1-FA-A# show fault

    Severity  Code     Last Transition Time     ID       Description

    --------- -------- ------------------------ -------- -----------

    Minor     F0476    2012-05-09T20:21:15.985   1668798 FC pool node-wwn-assignment node-default is empty

    Major     F0282    2012-02-08T13:58:11.915   1576231 lan port-channel 2 on fabric interconnect A oper state: failed, reason: No operational members

    Minor     F0465    2012-04-12T16:52:38.892    193850 IP pool iscsi-initiator-pool is empty

     

    Notice that the port channel fault on Fabric B is now not there in show fault command output.

    Clear "Port Channel with No Operational Member Fault" on Fabric Interconnect A
    
    UCS1-FA-A# scope eth-uplink
    UCS1-FA-A /eth-uplink # scope fabric a
    UCS1-FA-A /eth-uplink/fabric # enter port-channel 2
    UCS1-FA-A /eth-uplink/fabric/port-channel # disable
    UCS1-FA-A /eth-uplink/fabric/port-channel* # commit-buffer
    UCS1-FA-A /eth-uplink/fabric/port-channel # exit
    UCS1-FA-A /eth-uplink/fabric # exit
    UCS1-FA-A /eth-uplink # exit
    UCS1-FA-A# show fault
    
    Severity  Code     Last Transition Time     ID       Description
    
    --------- -------- ------------------------ -------- -----------
    
    Cleared  F0282    2012-05-11T15:54:27.840   1576231 lan port-channel 2 on fabric interconnect A oper state: failed, reason: No operational members
    
    Minor     F0465    2012-04-12T16:52:38.892    193850 IP pool iscsi-initiator-pool is empty
    Minor     F0476    2012-05-09T20:21:15.985   1668798 FC pool node-wwn-assignment node-default is empty

     

    Notice that the port channel fault on Fabric A shows "Cleared".

     

    Clear "IP Pool iscsi-initiator-pool empty" Fault
    
    UCS1-FA-A# scope org
    UCS1-FA-A /org # scope ip-pool iscsi-initiator-pool
    UCS1-FA-A /org/ip-pool # show
    IP Pool:
        Name                 Size       Assigned
    
        -------------------- ---------- --------
    
        iscsi-initiator-pool          0          0
    
    UCS1-FA-A /org/ip-pool # create block 1.1.1.1 1.1.1.1 1.1.1.1 255.255.255.0
    UCS1-FA-A /org/ip-pool/block* # commit-buffer
    UCS1-FA-A /org/ip-pool/block # exit
    UCS1-FA-A /org/ip-pool # show
    
    IP Pool:
    
        Name                 Size       Assigned
    
        -------------------- ---------- --------
    
        iscsi-initiator-pool          1        0
    
    UCS1-FA-A /org/ip-pool # show block
    
    Block of IP Addresses:
    
        From            To              Default Gateway Subnet Mask
        --------------- --------------- --------------- -----------
    
        1.1.1.1         1.1.1.1         1.1.1.1         255.255.255.0
    
    UCS1-FA-A /org/ip-pool # exit
    UCS1-FA-A /org # exit
    UCS1-FA-A# show fault
    
    Severity  Code     Last Transition Time     ID       Description
    --------- -------- ------------------------ -------- -----------
    Cleared  F0465    2012-05-11T17:37:52.178    193850 IP pool iscsi-initiator-pool is empty
    
    Minor     F0476    2012-05-09T20:21:15.985   1668798 FC pool node-wwn-assignment node-default is empty
    
    Notice that the IP pool iscsi-initiator-pool empty fault shows "Cleared".
    
    Clear "FC Pool node-default empty" Fault
    
    UCS1-FA-A /org # show wwn-pool
    
    WWN Pool:
    
        Name                 Purpose             Size       Assigned
        -------------------- ------------------- ---------- --------
        node-default         Node Wwn Assignment          0        0
    
    UCS1-FA-A /org # scope wwn-pool node-default
    UCS1-FA-A /org/wwn-pool # create initiator 50:00:25:B5:11:11:11:11
    UCS1-FA-A /org/wwn-pool/initiator* # commit-buffer
    UCS1-FA-A /org/wwn-pool/initiator # exit
    UCS1-FA-A /org/wwn-pool # exit
    UCS1-FA-A /org # exit
    UCS1-FA-A# show fault
    
    Severity  Code     Last Transition Time     ID       Description
    
    --------- -------- ------------------------ -------- -----------
    Cleared  F0476    2012-05-11T17:46:07.400   1668798 FC pool node-wwn-assignment node-default is empty
    Cleared   F0465    2012-05-11T17:37:52.178    193850 IP pool iscsi-initiator-pool is empty

    Notice that the FC pool node-default empty fault shows "Cleared".

    Related Information

    This widget could not be displayed.