C-Series // Resolve Service Profile Association Failure When Incorrect Server Info Given

[Tray Stoutmeyer]

C-Series Integrated Server // Troubleshooting and Resolving Service Profile Association Failure When Incorrect Server Information is Given

This article will walk you through how to resolve a F0327 error on a UCSM managed rack server where there is a failure to associate and the customer has given you incorrect information as to what server they were trying to associate.

Problem

A customer called in with a problem associating a service profile to a server. They receive the following error when attempting to associate service profile STR_38 with a server.

Error Code: F0327

Cause: Configuration Failure

Description: Service profile STR_38 configuration failed due to insufficient-resources, connection-placement, insufficient-vnic-capacity.

I gathered the necessary UCSM logs and the customer told me they were trying to associate server 57 which was incorrect information. When I could see Server 57 was already associated with a different service profile, I didn't want to bother the customer with asking more questions and tracked it down myself so we could resolve the issue faster.

Solution

Customer errors from attempted service profile association of service profile STR_38:

Once you have UCSM logs, you can open the UCSM and expand the A side and B side zip files. Once this is done, open the SAM_TECHSUPPORT file on whichever unzipped A or B side has it open it in a text editor like Notepad++ (mac/pc) or TextWrangler (mac). Since we know the service profile name (in this case STR_38), search for the following phrase and search for all occurrences of it.

"STR_38 failed"

You will be jumped to the following minor errors and warnings telling you the following:

• Both when and why it failed.
• What portion of the service profile actually failed that would lead to insufficient resources (although the error already points to possible vnic issues)
• What server it was trying to associate.

Severity: Minor
Code: F0169
Last Transition Time: 2016-07-07T11:43:04.671
ID: 3727100
Status: None
Description: Eth vNIC vNIC1, service profile STR_38 failed to apply configuration
Affected Object: org-root/org-STR_DDS/ls-STR_38/ether-vNIC1
Name: Vnic Ether Config Failed
Cause: Configuration Failed
Type: Configuration
Acknowledged: No
Occurrences: 1
Creation Time: 2016-07-07T11:43:04.671
Original Severity: Minor
Previous Severity: Minor
Highest Severity: Minor

Severity: Minor
Code: F0169
Last Transition Time: 2016-07-07T11:43:04.670
ID: 3727099
Status: None
Description: Eth vNIC vNIC0, service profile STR_38 failed to apply configuration
Affected Object: org-root/org-STR_DDS/ls-STR_38/ether-vNIC0
Name: Vnic Ether Config Failed
Cause: Configuration Failed
Type: Configuration
Acknowledged: No
Occurrences: 1
Creation Time: 2016-07-07T11:43:04.670
Original Severity: Minor
Previous Severity: Minor
Highest Severity: Minor

Severity: Major
Code: F0327
Last Transition Time: 2016-07-07T11:43:04.669
ID: 3727098
Status: None
Description: Service profile STR_38 configuration failed due to insufficient-resources,connection-placement,insufficient-vnic-capacity
Affected Object: org-root/org-STR_DDS/ls-STR_38
Name: Ls Server Config Failure
Cause: Configuration Failure
Type: Server
Acknowledged: No
Occurrences: 1
Creation Time: 2016-07-07T11:43:04.669
Original Severity: Major
Previous Severity: Major
Highest Severity: Major

Severity: Minor
Code: F0689
Last Transition Time: 2016-07-07T11:43:04.668
ID: 3727096
Status: None
Description: Assignment of service profile STR_38 to server sys/rack-unit-47 failed <=== Rack Server 47
Affected Object: org-root/org-STR_DDS/ls-STR_38/pn
Name: Ls Compute Binding Assignment Requirements Not Met
Cause: Assignment Failed
Type: Server
Acknowledged: No
Occurrences: 1
Creation Time: 2016-07-07T11:43:04.668
Original Severity: Minor
Previous Severity: Minor
Highest Severity: Minor

You can see this failed on the VNIC portion.

Using this information you can now search the same SAM_TECHSUPPORT file to look for all occurrences of Server 47.

The first hit on the list will take you to inventory listing of Server 47 here that will include the type of VIC (where your NICs are located). The VIC info is the last line of the inventory output.

Server 47:
    Name:
    Model: UCSC-C240-M4SX
    Acknowledged Vendor: Cisco Systems Inc
    Acknowledged Serial (SN): XXXXXXXX
    Acknowledged Revision: 0
    Acknowledged Product Name: Cisco UCS C240 M4SX
    Acknowledged PID: UCSC-C240-M4SX
    Acknowledged VID: 0
    Acknowledged Memory (MB): 131072
    Acknowledged Effective Memory (MB): 131072
    Acknowledged Cores: 16
    Acknowledged Adapters: 1
    Acknowledged Burned-In UUID: f98ebd24-5287-4d80-adf0-d04cdb359dd4
    Acknowledged Dynamic UUID: f98ebd24-5287-4d80-adf0-d04cdb359dd4

<======Information redacted to save space==================>

Adapter:

        Adapter PID          Vendor            Serial       Overall Status
        ------- ------------ ----------------- ------------ --------------
              1 UCSC-MLOM-CSC-02
                             Cisco Systems Inc FCH19447AQR  N/A

UCSC-MLOM-CSC-02 is a VIC 1227

The second hit on the list for Server 47 will take you to show you the connectivity.

Server 47:
    Conn Path: A,B
    Conn Status: A
    Managing Instance: A
    Availability: Available
    Admin State: In Service
    Overall Status: Unassociated
    Oper Qualifier: N/A
    Discovery: Complete
    Current Task:
    Check Point: Discovered

You will notice that the Conn Path is set for A,B but the Conn Status is only A. Instead of jumping to conclusions I checked other rack servers in the same area on the same log and all are set to A/B.

Server 45:
    Conn Path: A,B
    Conn Status: A,B
    Managing Instance: A
    Availability: Unavailable
    Admin State: In Service
    Overall Status: Ok
    Oper Qualifier: N/A
    Discovery: Complete
    Current Task:
    Check Point: Discovered

Server 46:
    Conn Path: A,B
    Conn Status: A,B
    Managing Instance: B
    Availability: Unavailable
    Admin State: In Service
    Overall Status: Ok
    Oper Qualifier: N/A
    Discovery: Complete
    Current Task:
    Check Point: Discovered

To triple check this a little further you can look for "scope server 47" on the same SAM_TECHSUPPORT file and the second hit will take you to the adapter details which will also reflect incorrect connectivity.

`scope server 47`
`scope adapter 1`
`show expand detail`

Adapter:
    Id: 1
    Product Name: Cisco UCS MLOM 1227
    PID: UCSC-MLOM-CSC-02
    VID: V00
    Vendor: Cisco Systems Inc
    Serial: XXXXXXXX
    Revision: 0
    Mfg Date: 2015-10-29T01:00:00.000
    Slot: MLOM
    Overall Status: N/A
    Conn Path: A
    Conn Status: A
    Managing Instance: A
    Product Description: PCI Express Dual Port Cisco Converged Network Adapter Supporting 10Gb Ethernet and 10Gb Fibre Channel

    Ext Interface:
        Mezz Id: 1
        Id: 1
        Port Channel:
        Mac: 58:AC:78:5C:BF:2C
        If Type: Physical
        Link Status: Up
        Oper State: Link Up
        Oper State Desc:
        Purpose: General
        Side: Left
        Slot Id: 1
        Fabric: NONE
        Peer Port Id: 0
        Peer Slot Id: 0
        Peer Chassis Id: N/A
        Current Task:

        Mezz Id: 1
        Id: 2
        Port Channel:
        Mac: 58:AC:78:5C:BF:2D
        If Type: Physical
        Link Status: Up
        Oper State: Link Up
        Oper State Desc:
        Purpose: General
        Side: Right
        Slot Id: 1
        Fabric: A
        Peer Port Id: 27
        Peer Slot Id: 1
        Peer Chassis Id: 3
        Current Task:

The giveaway there is only one side connectivity is that one side says Fabric A and the other side says NONE.

I know some of you are probably saying "Why not just go look at the VIF paths when you see the VNIC errors and also the main error on UCSM stating possible VNIC configuration?" The customer was looking at the wrong server and saw all VIF paths up when I asked him to look. This led us here.

You can see the same in the SAM_TECHSUPPORT for server 57 here. Shows both paths.

`scope server 57`
`scope adapter 1`
`show expand detail`

Adapter:
    Id: 1
    Product Name: Cisco UCS MLOM 1227
    PID: UCSC-MLOM-CSC-02
    VID: V00
    Vendor: Cisco Systems Inc
    Serial: XXXXXXX
    Revision: 0
    Mfg Date: 2015-09-11T01:00:00.000
    Slot: MLOM
    Overall Status: N/A
    Conn Path: A,B
    Conn Status: A,B
    Managing Instance: A
    Product Description: PCI Express Dual Port Cisco Converged Network Adapter Supporting 10Gb Ethernet and 10Gb Fibre Channel

    Ext Interface:
        Mezz Id: 1
        Id: 1
        Port Channel:
        Mac: E4:AA:5D:42:3E:93
        If Type: Physical
        Link Status: Up
        Oper State: Link Up
        Oper State Desc:
        Purpose: General
        Side: Left
        Slot Id: 1
        Fabric: B
        Peer Port Id: 25
        Peer Slot Id: 1
        Peer Chassis Id: 4
        Current Task:

        Mezz Id: 1
        Id: 2
        Port Channel:
        Mac: E4:AA:5D:42:3E:94
        If Type: Physical
        Link Status: Up
        Oper State: Link Up
        Oper State Desc:
        Purpose: General
        Side: Right
        Slot Id: 1
        Fabric: A
        Peer Port Id: 25
        Peer Slot Id: 1
        Peer Chassis Id: 3
        Current Task:

You can see how the customer telling you and looking at the wrong server on UCSM can lead you to the wrong path and some head scratching.

So once I had the right information, I asked the customer to check the VIF paths tab after selecting server 47 and sure enough only the A path was up causing the association of the service profile that wanted/needed two paths to fail with insufficient resources. Once they corrected the B path side connectivity issues, the service profile associated just fine.

I hope this will not only help you in this type of situation but I also hope that it's a good reference on how to check what causes a service profile to fail association and check connectivity that exists on server hardware.

Thanks!

Tray

[Tray Stoutmeyer]

C-Series Integrated Server // Troubleshooting and Resolving Service Profile Association Failure When Incorrect Server Information is Given

This article will walk you through how to resolve a F0327 error on a UCSM managed rack server where there is a failure to associate and the customer has given you incorrect information as to what server they were trying to associate.

Problem

A customer called in with a problem associating a service profile to a server. They receive the following error when attempting to associate service profile STR_38 with a server.

Error Code: F0327

Cause: Configuration Failure

Description: Service profile STR_38 configuration failed due to insufficient-resources, connection-placement, insufficient-vnic-capacity.

I gathered the necessary UCSM logs and the customer told me they were trying to associate server 57 which was incorrect information. When I could see Server 57 was already associated with a different service profile, I didn't want to bother the customer with asking more questions and tracked it down myself so we could resolve the issue faster.

Solution

Customer errors from attempted service profile association of service profile STR_38:

Once you have UCSM logs, you can open the UCSM and expand the A side and B side zip files. Once this is done, open the SAM_TECHSUPPORT file on whichever unzipped A or B side has it open it in a text editor like Notepad++ (mac/pc) or TextWrangler (mac). Since we know the service profile name (in this case STR_38), search for the following phrase and search for all occurrences of it.

"STR_38 failed"

You will be jumped to the following minor errors and warnings telling you the following:

• Both when and why it failed.
• What portion of the service profile actually failed that would lead to insufficient resources (although the error already points to possible vnic issues)
• What server it was trying to associate.

Severity: Minor
Code: F0169
Last Transition Time: 2016-07-07T11:43:04.671
ID: 3727100
Status: None
Description: Eth vNIC vNIC1, service profile STR_38 failed to apply configuration
Affected Object: org-root/org-STR_DDS/ls-STR_38/ether-vNIC1
Name: Vnic Ether Config Failed
Cause: Configuration Failed
Type: Configuration
Acknowledged: No
Occurrences: 1
Creation Time: 2016-07-07T11:43:04.671
Original Severity: Minor
Previous Severity: Minor
Highest Severity: Minor

Severity: Minor
Code: F0169
Last Transition Time: 2016-07-07T11:43:04.670
ID: 3727099
Status: None
Description: Eth vNIC vNIC0, service profile STR_38 failed to apply configuration
Affected Object: org-root/org-STR_DDS/ls-STR_38/ether-vNIC0
Name: Vnic Ether Config Failed
Cause: Configuration Failed
Type: Configuration
Acknowledged: No
Occurrences: 1
Creation Time: 2016-07-07T11:43:04.670
Original Severity: Minor
Previous Severity: Minor
Highest Severity: Minor

Severity: Major
Code: F0327
Last Transition Time: 2016-07-07T11:43:04.669
ID: 3727098
Status: None
Description: Service profile STR_38 configuration failed due to insufficient-resources,connection-placement,insufficient-vnic-capacity
Affected Object: org-root/org-STR_DDS/ls-STR_38
Name: Ls Server Config Failure
Cause: Configuration Failure
Type: Server
Acknowledged: No
Occurrences: 1
Creation Time: 2016-07-07T11:43:04.669
Original Severity: Major
Previous Severity: Major
Highest Severity: Major

Severity: Minor
Code: F0689
Last Transition Time: 2016-07-07T11:43:04.668
ID: 3727096
Status: None
Description: Assignment of service profile STR_38 to server sys/rack-unit-47 failed <=== Rack Server 47
Affected Object: org-root/org-STR_DDS/ls-STR_38/pn
Name: Ls Compute Binding Assignment Requirements Not Met
Cause: Assignment Failed
Type: Server
Acknowledged: No
Occurrences: 1
Creation Time: 2016-07-07T11:43:04.668
Original Severity: Minor
Previous Severity: Minor
Highest Severity: Minor

You can see this failed on the VNIC portion.

Using this information you can now search the same SAM_TECHSUPPORT file to look for all occurrences of Server 47.

The first hit on the list will take you to inventory listing of Server 47 here that will include the type of VIC (where your NICs are located). The VIC info is the last line of the inventory output.

Server 47:
    Name:
    Model: UCSC-C240-M4SX
    Acknowledged Vendor: Cisco Systems Inc
    Acknowledged Serial (SN): XXXXXXXX
    Acknowledged Revision: 0
    Acknowledged Product Name: Cisco UCS C240 M4SX
    Acknowledged PID: UCSC-C240-M4SX
    Acknowledged VID: 0
    Acknowledged Memory (MB): 131072
    Acknowledged Effective Memory (MB): 131072
    Acknowledged Cores: 16
    Acknowledged Adapters: 1
    Acknowledged Burned-In UUID: f98ebd24-5287-4d80-adf0-d04cdb359dd4
    Acknowledged Dynamic UUID: f98ebd24-5287-4d80-adf0-d04cdb359dd4

<======Information redacted to save space==================>

Adapter:

        Adapter PID          Vendor            Serial       Overall Status
        ------- ------------ ----------------- ------------ --------------
              1 UCSC-MLOM-CSC-02
                             Cisco Systems Inc FCH19447AQR  N/A

UCSC-MLOM-CSC-02 is a VIC 1227

The second hit on the list for Server 47 will take you to show you the connectivity.

Server 47:
    Conn Path: A,B
    Conn Status: A
    Managing Instance: A
    Availability: Available
    Admin State: In Service
    Overall Status: Unassociated
    Oper Qualifier: N/A
    Discovery: Complete
    Current Task:
    Check Point: Discovered

You will notice that the Conn Path is set for A,B but the Conn Status is only A. Instead of jumping to conclusions I checked other rack servers in the same area on the same log and all are set to A/B.

Server 45:
    Conn Path: A,B
    Conn Status: A,B
    Managing Instance: A
    Availability: Unavailable
    Admin State: In Service
    Overall Status: Ok
    Oper Qualifier: N/A
    Discovery: Complete
    Current Task:
    Check Point: Discovered

Server 46:
    Conn Path: A,B
    Conn Status: A,B
    Managing Instance: B
    Availability: Unavailable
    Admin State: In Service
    Overall Status: Ok
    Oper Qualifier: N/A
    Discovery: Complete
    Current Task:
    Check Point: Discovered

To triple check this a little further you can look for "scope server 47" on the same SAM_TECHSUPPORT file and the second hit will take you to the adapter details which will also reflect incorrect connectivity.

`scope server 47`
`scope adapter 1`
`show expand detail`

Adapter:
    Id: 1
    Product Name: Cisco UCS MLOM 1227
    PID: UCSC-MLOM-CSC-02
    VID: V00
    Vendor: Cisco Systems Inc
    Serial: XXXXXXXX
    Revision: 0
    Mfg Date: 2015-10-29T01:00:00.000
    Slot: MLOM
    Overall Status: N/A
    Conn Path: A
    Conn Status: A
    Managing Instance: A
    Product Description: PCI Express Dual Port Cisco Converged Network Adapter Supporting 10Gb Ethernet and 10Gb Fibre Channel

    Ext Interface:
        Mezz Id: 1
        Id: 1
        Port Channel:
        Mac: 58:AC:78:5C:BF:2C
        If Type: Physical
        Link Status: Up
        Oper State: Link Up
        Oper State Desc:
        Purpose: General
        Side: Left
        Slot Id: 1
        Fabric: NONE
        Peer Port Id: 0
        Peer Slot Id: 0
        Peer Chassis Id: N/A
        Current Task:

        Mezz Id: 1
        Id: 2
        Port Channel:
        Mac: 58:AC:78:5C:BF:2D
        If Type: Physical
        Link Status: Up
        Oper State: Link Up
        Oper State Desc:
        Purpose: General
        Side: Right
        Slot Id: 1
        Fabric: A
        Peer Port Id: 27
        Peer Slot Id: 1
        Peer Chassis Id: 3
        Current Task:

The giveaway there is only one side connectivity is that one side says Fabric A and the other side says NONE.

I know some of you are probably saying "Why not just go look at the VIF paths when you see the VNIC errors and also the main error on UCSM stating possible VNIC configuration?" The customer was looking at the wrong server and saw all VIF paths up when I asked him to look. This led us here.

You can see the same in the SAM_TECHSUPPORT for server 57 here. Shows both paths.

`scope server 57`
`scope adapter 1`
`show expand detail`

Adapter:
    Id: 1
    Product Name: Cisco UCS MLOM 1227
    PID: UCSC-MLOM-CSC-02
    VID: V00
    Vendor: Cisco Systems Inc
    Serial: XXXXXXX
    Revision: 0
    Mfg Date: 2015-09-11T01:00:00.000
    Slot: MLOM
    Overall Status: N/A
    Conn Path: A,B
    Conn Status: A,B
    Managing Instance: A
    Product Description: PCI Express Dual Port Cisco Converged Network Adapter Supporting 10Gb Ethernet and 10Gb Fibre Channel

    Ext Interface:
        Mezz Id: 1
        Id: 1
        Port Channel:
        Mac: E4:AA:5D:42:3E:93
        If Type: Physical
        Link Status: Up
        Oper State: Link Up
        Oper State Desc:
        Purpose: General
        Side: Left
        Slot Id: 1
        Fabric: B
        Peer Port Id: 25
        Peer Slot Id: 1
        Peer Chassis Id: 4
        Current Task:

        Mezz Id: 1
        Id: 2
        Port Channel:
        Mac: E4:AA:5D:42:3E:94
        If Type: Physical
        Link Status: Up
        Oper State: Link Up
        Oper State Desc:
        Purpose: General
        Side: Right
        Slot Id: 1
        Fabric: A
        Peer Port Id: 25
        Peer Slot Id: 1
        Peer Chassis Id: 3
        Current Task:

You can see how the customer telling you and looking at the wrong server on UCSM can lead you to the wrong path and some head scratching.

So once I had the right information, I asked the customer to check the VIF paths tab after selecting server 47 and sure enough only the A path was up causing the association of the service profile that wanted/needed two paths to fail with insufficient resources. Once they corrected the B path side connectivity issues, the service profile associated just fine.

I hope this will not only help you in this type of situation but I also hope that it's a good reference on how to check what causes a service profile to fail association and check connectivity that exists on server hardware.

Thanks!

Tray