cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3010
Views
0
Helpful
14
Replies

I get “INTERFACE_QUARANTINED due to Cmd Failure” error when enabling PortChannel in n1kv

orevilla
Level 1
Level 1

I have added a new host to my n1kv, but the interface PortChannel3 appears to be in shutdown and I can't bring it up, despite of it is working.

SW-OUT-N1KV# sh int status

[...]

Po3            --                 up       trunk     full    1000    --

[...]

SW-OUT-N1KV# sh port-channel sum

Flags:  D - Down        P - Up in port-channel (members)

        I - Individual  H - Hot-standby (LACP only)

        s - Suspended   r - Module-removed

        S - Switched    R - Routed

        U - Up (port-channel)

--------------------------------------------------------------------------------

Group Port-       Type     Protocol  Member Ports

      Channel

--------------------------------------------------------------------------------

1     Po1(SD)     Eth      NONE      Eth3/2(r)

2     Po2(SU)     Eth      LACP      Eth4/5(P)    Eth4/6(P)

3     Po3(SU)     Eth      LACP      Eth5/5(P)    Eth5/6(P)

4     Po4(SU)     Eth      LACP      Eth4/1(P)    Eth4/2(P)    Eth4/3(P)

                                     Eth4/4(P)

5     Po5(SD)     Eth      LACP      Eth5/1(D)    Eth5/2(D)    Eth5/3(D)

                                     Eth5/4(D)

6     Po6(SD)     Eth      NONE      --

SW-OUT-N1KV# sh run int po3

!Command: show running-config interface port-channel3

!Time: Thu Jan  5 18:00:08 2012

version 4.2(1)SV1(4)

interface port-channel3

  inherit port-profile SYSTEM-UPLINK

  shutdown

When I try to enable it, I get the following error:

SW-OUT-N1KV(config)# int po3

SW-OUT-N1KV(config-if)# no shut

2012 Jan  5 18:00:51 SW-OUT-N1KV %PORT-PROFILE-2-INTERFACE_QUARANTINED: Interface port-channel3 has been quarantined due to Cmd Failure

The "show logging logfile" and "show accounting log" shows nothing.

Any idea why is it happening????

Other outputs,

SW-OUT-N1KV(config-if)# do sh run port-profile SYSTEM-UPLINK

!Command: show running-config port-profile SYSTEM-UPLINK

!Time: Thu Jan  5 18:02:46 2012

version 4.2(1)SV1(4)

port-profile type ethernet SYSTEM-UPLINK

  vmware port-group

  switchport mode trunk

  duplex auto

  speed auto

  switchport trunk native vlan 666

  switchport trunk allowed vlan 2,43-44

  cdp enable

  channel-group auto mode active

  no shutdown

  system vlan 2,43-44

  state enabled

SW-OUT-N1KV(config-if)# do sh port-prof virt usag

-------------------------------------------------------------------------------

Port Profile               Port        Adapter        Owner

-------------------------------------------------------------------------------

SYSTEM-UPLINK              Po1

                           Po2

                           Po3

                           Eth4/5      vmnic4         10.100.100.63

                           Eth4/6      vmnic5         10.100.100.63

                           Eth5/5      vmnic4         10.100.100.64

                           Eth5/6      vmnic5         10.100.100.64

IESE_CONTENIDO             Veth9       Net Adapter 1  IESECMPRO

                           Veth14      Net Adapter 1  IESEAUTOPRO

IESE_BACKEND               Veth10      Net Adapter 1  IESEDBINTPRE

                           Veth11      Net Adapter 1  IESEDBPRO

IESE_PRE_INT               Veth3       Net Adapter 1  IESEWEBPRE

                           Veth4       Net Adapter 1  IESEAPPINT

                           Veth5       Net Adapter 1  IESEAPPPRE1

                           Veth6       Net Adapter 1  IESEAPPPRE2

                           Veth7       Net Adapter 1  IESEAUTOPRE

                           Veth8       Net Adapter 1  IESECMINTPRE

DATA-UPLINK-100m           Po6

SERVICE-CONSOLE            Veth1       vmk0           Module 4

                           Veth2       vmk1           Module 4

                           Veth12      vmk0           Module 5

                           Veth13      vmk1           Module 5

DATA-UPLINK-IESE           Po4

                           Po5

                           Eth4/1      vmnic0         10.100.100.63

                           Eth4/2      vmnic1         10.100.100.63

                           Eth4/3      vmnic2         10.100.100.63

                           Eth4/4      vmnic3         10.100.100.63

                           Eth5/1      vmnic0         10.100.100.64

                           Eth5/2      vmnic1         10.100.100.64

                           Eth5/3      vmnic2         10.100.100.64

                           Eth5/4      vmnic3         10.100.100.64

1 Accepted Solution

Accepted Solutions

Hello Oscar,

Please open a TAC service request to further troubleshoot this issue.

Following additional info would help here

Is this new configuration  ?

Are the uplinks connecting to same physical switch or different switches ?

Is this issue only specific to this port channel ?

Padma

View solution in original post

14 Replies 14

cparik
Level 1
Level 1

Hello Oscar,

This generally happens if a command on the port profile failed to be appllied.

Can you send the output of the "show logging logfile | grep INTERFACE_CMD_FAILURE"  and "show accounting log"?

You can look for the failed commands on the quarantined interfaces.

Thanks,

Chetan

Hi Chetan,

The first one shows nothing,

SW-OUT-N1KV# show logging logfile | grep INTERFACE_CMD_FAILURE

SW-OUT-N1KV#

And the second one shows the following,

Mon Jan  9 12:01:38 2012:update:192.168.237.49@pts/3:admin:configure terminal ; interface port-channel3 (SUCCESS)

Mon Jan  9 12:01:41 2012:update:192.168.237.49@pts/3:admin:configure terminal ; interface port-channel3 ; no shutdown (REDIRECT)

Mon Jan  9 12:01:41 2012:update:

192.168.237.49@pts/3:admin:configure

terminal ; interface port-channel3 (SUCCESS)

Mon Jan  9 12:01:41 2012:update:

192.168.237.49@pts/3:admin:configure

terminal ; interface port-channel3 ; no shutdown (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; switchport mode trunk (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; duplex auto (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; speed auto (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; switchport trunk native vlan 666 (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; switchport trunk allowed vlan 2, 43-44 (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.873:admin:configure terminal ; cdp enable (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.883:admin:configure terminal ; interface port-channel3 (SUCCESS)

Mon Jan  9 12:01:42 2012:update:ppm.883:admin:configure terminal ; interface port-channel3 ; shutdown (FAILURE)

Thanks,

Debugging port-profile errors, and trying agair, I get the following output:

SW-OUT-N1KV(config-port-prof)# no shut

2012 Jan 9 14:43:02.102017 port-profile: (ERR) ppm_fq_cmd_intf_range_get(1753): Failed to find intf tlv ^Yex^Yconf^Yport-profile DATA-UPLINK-IESE-SA^Yno shutdown error: Command Parsing Failed

2012 Jan 9 14:43:02.102577 port-profile: (ERR) ppm_profile_fsm_session_get(71): RID before session start request is 1

2012 Jan 9 14:43:02.106573 port-profile: (ERR) ppm_req_dbmgr_send(426): Changing session status: 0x0

2012 Jan 9 14:43:02.443092 port-profile: (ERR) ppm_vppm_profile_acfg_gen(353): VPPM Profile Level -1 acfg generation failed Error: no such pss key (0x40480003)

2012 Jan 9 14:43:03.048365 port-profile: (ERR) ppm_pending_req_queue_check(1035): msg_ref_p->state == 0xbba6e3cc

SW-OUT-N1KV(config-port-prof)# 2012 Jan 9 14:43:03.373461 port-profile: (ERR) ppm_pending_req_queue_check(1035): msg_ref_p->state == 0xbba6e3cc

SW-OUT-N1KV(config-port-prof)# channel-group auto mode active

2012 Jan 9 15:37:32.338832 port-profile: (ERR) ppm_fq_cmd_intf_range_get(1753): Failed to find intf tlv ^Yex^Yconf^Yport-profile type ethernet DATA-UPLINK-IESE-SA^Ychannel-group auto mode active error: Command Parsing Failed

2012 Jan 9 15:37:32.339399 port-profile: (ERR) ppm_profile_fsm_session_get(71): RID before session start request is 1

2012 Jan 9 15:37:32.358546 port-profile: (ERR) ppm_db_port_channel_check(7068): Not a channel grp command.

2012 Jan 9 15:37:32.360130 port-profile: (ERR) ppm_req_dbmgr_send(426): Changing session status: 0x0

2012 Jan 9 15:37:32.651020 port-profile: (ERR) ppm_vppm_profile_acfg_gen(353): VPPM Profile Level -1 acfg generation failed Error: no such pss key (0x40480003)

2012 Jan 9 15:37:33.181682 port-profile: (ERR) ppm_pending_req_queue_check(1035): msg_ref_p->state == 0xbba6e3cc

2012 Jan 9 15:37:33.387996 port-profile: (ERR) ppm_find_intf_in_cache(917): No pending plan present

2012 Jan 9 15:37:34.772375 port-profile: (ERR) ppm_pending_req_queue_check(1035): msg_ref_p->state == 0xbba6e3cc

2012 Jan 9 15:37:34.774586 port-profile: (ERR) ppm_profile_fsm_session_get(71): RID before session start request is 1

SW-OUT-N1KV(config-port-prof)# 2012 Jan 9 15:37:35.102895 port-profile: (ERR) ppm_vppm_profile_acfg_gen(353): VPPM Profile Level -1 acfg generation failed Error: no such pss key (0x40480003)

2012 Jan 9 15:37:35.469793 port-profile: (ERR) ppm_config_merge_handler(773): Show run merge not performed with reason PPM show run merge is not required

2012 Jan 9 15:37:35.502211 port-profile: (ERR) ppm_cmd_desc_create(1176): command to be ignored parsed mode:/exec/configuremode: /exec/configure cmd:version 4.2(1)SV1(4) error: Command is not a port-profile command

2012 Jan 9 15:37:35.503417 port-profile: (ERR) ppm_cmd_desc_create(1265): interface command detected ^Yex^Yconf^Yinterface port-channel5, error: Interface command is given for cmd desciptor

2012 Jan 9 15:37:35.510826 port-profile: (ERR) ppm_req_dbmgr_send(426): Changing session status: 0x0

2012 Jan 9 15:37:59.800813 port-profile: (ERR) ppm_dispatcher_invoker(542): Invoker parent failed with Err[0x1400]

2012 Jan 9 15:37:59.832496 port-profile: (ERR) procjobcb_job_done(686): status: 0x1400, (null)

2012 Jan 9 15:37:59.837319 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1250): Changing session status: 0x420c007c

2012 Jan 9 15:37:59.837732 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1252): apply status: 0x1400

2012 Jan 9 15:37:59.838435 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1277): Could not open the file /dev/shm/ppm_DATA-UPLINK-IESE-SA_output.txt

2012 Jan 9 15:37:59.800813 port-profile: (ERR) ppm_dispatcher_invoker(542): Invoker parent failed with Err[0x1400]

2012 Jan 9 15:37:59.832496 port-profile: (ERR) procjobcb_job_done(686): status: 0x1400, (null)

2012 Jan 9 15:37:59.837319 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1250): Changing session status: 0x420c007c

2012 Jan 9 15:37:59.837732 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1252): apply status: 0x1400

2012 Jan 9 15:37:59.838435 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1277): Could not open the file /dev/shm/ppm_DATA-UPLINK-IESE-SA_output.txt

2012 Jan 9 15:37:59 SW-OUT-N1KV %PORT-PROFILE-2-INTERFACE_QUARANTINED: Interface port-channel5 has been quarantined due to Cmd Failure

Any idea? I'm lost...

Hello Oscar,

Please open a TAC service request to further troubleshoot this issue.

Following additional info would help here

Is this new configuration  ?

Are the uplinks connecting to same physical switch or different switches ?

Is this issue only specific to this port channel ?

Padma

I'm sorry. I clicked on "correct answer" by mistake.

The same port-profile is working fine on other interfaces on other hosts. I'm assuming there is no problem having different hosts using the same port-profile.

The uplinks are connecting to two upsteam physical stacked switches.

I have done several tests and the results are quite confusing, at least for me.

All the interfaces on host A are working fine in PortChannel using port-profile DATA-UPLINK-IESE

All the interfaces on host B are working fine in PortChannel using port-profile DATA-UPLINK-IESE-SA

Upsteam switches' configuration are equal in both Port-Channels.

SW-OUT-N1KV# sh run port-prof DATA-UPLINK-IESE

!Command: show running-config port-profile DATA-UPLINK-IESE
!Time: Tue Jan 10 18:22:51 2012

version 4.2(1)SV1(4)
port-profile type ethernet DATA-UPLINK-IESE
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 85-86,88
  switchport trunk native vlan 666
  speed auto
  duplex auto
  cdp enable
  channel-group auto mode active
  no shutdown
  state enabled


SW-OUT-N1KV# sh run port-prof DATA-UPLINK-IESE-SA

!Command: show running-config port-profile DATA-UPLINK-IESE-SA
!Time: Tue Jan 10 18:23:03 2012

version 4.2(1)SV1(4)
port-profile type ethernet DATA-UPLINK-IESE-SA
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 85-86,88
  switchport trunk native vlan 666
  speed auto
  duplex auto
  cdp enable
  channel-group auto mode active
  no shutdown
  state enabled

When I try to bind PNICs on host A to the port-profile DATA-UPLINK-IESE-SA, the PortChannel is quarantined.

The same, when I try to bind interfaces on host B to the port-profile DATA-UPLINK-IESE.

I have solved the issue, creating a new port-profile, copying the configuration from the above ones, and binding all the interfaces on hosts A and B to that new port-profile.

SW-OUT-N1KV# sh run port-prof DATA-UPLINK-PROV

!Command: show running-config port-profile DATA-UPLINK-PROV
!Time: Tue Jan 10 18:28:33 2012

version 4.2(1)SV1(4)
port-profile type ethernet DATA-UPLINK-PROV
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 85-86,88
  switchport trunk native vlan 666
  speed auto
  duplex auto
  channel-group auto mode active
  no shutdown
  state enabled

-------------------------------------------------------------------------------

Port Profile               Port        Adapter        Owner

-------------------------------------------------------------------------------

DATA-UPLINK-PROV           Po3

                           Po4

                           Eth4/1      vmnic0         10.100.100.63

                           Eth4/2      vmnic1         10.100.100.63

                           Eth4/3      vmnic2         10.100.100.63

                           Eth4/4      vmnic3         10.100.100.63

                           Eth5/1      vmnic0         10.100.100.64

                           Eth5/2      vmnic1         10.100.100.64

                           Eth5/3      vmnic2         10.100.100.64

                           Eth5/4      vmnic3         10.100.100.64

Do you have any idea why is it happening?

Thanks,

Oscar,

Mistakes do happen :-)

We need logs to better understand the behavior. If you have not done already,I would suggest you to open SR and upload the logs to better understand the behavior.

Padma

Yes, I am also opening a SR.

What kind of logs do you need? the "show log logfile"? or anything else more spcific?

Thanks,

Oscar,

To start with, we need VSM and VEM show tech bundle

VSM

show tech-support  svs | no-more

VEM ( SSh into ESXi host and execute the command )

vem-support all

Padma

Hi Padma,

I attach the outputs,

Thanks,

Oscar,

Thanks for the logs.

Once you have opened TAC SR, engineer would follow up with you on the analysis.

I will also check it out.

Padma

Thank you for everything Padramas,

the engineer is already asking me for some information.

I have realised that deleting the port-profile and re-creating it (with the same config) solves the problem. After that, I can bind the interfaces with the new port-profile and everything is fine. But there must be another way to remove this "state" in the port-profile without the need of deleting it completely.

Do you know any way to do so?

Thanks

Oscar,

Can you please send me a private message with your TAC SR number ?

Padma

Hello.

I've exactly the same problem in my production site.

Please, can you post the solution the TAC gave to you ? Or can you send it me in a private message ?

Thank you.

Hello Julien,

I could not find a TAC service request for this issue.

Please open one so that we can help you out.

Padma

Review Cisco Networking for a $25 gift card