01-09-2012 02:24 AM - edited 03-01-2019 10:13 AM
I have added a new host to my n1kv, but the interface PortChannel3 appears to be in shutdown and I can't bring it up, despite of it is working.
SW-OUT-N1KV# sh int status
[...]
Po3 -- up trunk full 1000 --
[...]
SW-OUT-N1KV# sh port-channel sum
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SD) Eth NONE Eth3/2(r)
2 Po2(SU) Eth LACP Eth4/5(P) Eth4/6(P)
3 Po3(SU) Eth LACP Eth5/5(P) Eth5/6(P)
4 Po4(SU) Eth LACP Eth4/1(P) Eth4/2(P) Eth4/3(P)
Eth4/4(P)
5 Po5(SD) Eth LACP Eth5/1(D) Eth5/2(D) Eth5/3(D)
Eth5/4(D)
6 Po6(SD) Eth NONE --
SW-OUT-N1KV# sh run int po3
!Command: show running-config interface port-channel3
!Time: Thu Jan 5 18:00:08 2012
version 4.2(1)SV1(4)
interface port-channel3
inherit port-profile SYSTEM-UPLINK
shutdown
When I try to enable it, I get the following error:
SW-OUT-N1KV(config)# int po3
SW-OUT-N1KV(config-if)# no shut
2012 Jan 5 18:00:51 SW-OUT-N1KV %PORT-PROFILE-2-INTERFACE_QUARANTINED: Interface port-channel3 has been quarantined due to Cmd Failure
The "show logging logfile" and "show accounting log" shows nothing.
Any idea why is it happening????
Other outputs,
SW-OUT-N1KV(config-if)# do sh run port-profile SYSTEM-UPLINK
!Command: show running-config port-profile SYSTEM-UPLINK
!Time: Thu Jan 5 18:02:46 2012
version 4.2(1)SV1(4)
port-profile type ethernet SYSTEM-UPLINK
vmware port-group
switchport mode trunk
duplex auto
speed auto
switchport trunk native vlan 666
switchport trunk allowed vlan 2,43-44
cdp enable
channel-group auto mode active
no shutdown
system vlan 2,43-44
state enabled
SW-OUT-N1KV(config-if)# do sh port-prof virt usag
-------------------------------------------------------------------------------
Port Profile Port Adapter Owner
-------------------------------------------------------------------------------
SYSTEM-UPLINK Po1
Po2
Po3
Eth4/5 vmnic4 10.100.100.63
Eth4/6 vmnic5 10.100.100.63
Eth5/5 vmnic4 10.100.100.64
Eth5/6 vmnic5 10.100.100.64
IESE_CONTENIDO Veth9 Net Adapter 1 IESECMPRO
Veth14 Net Adapter 1 IESEAUTOPRO
IESE_BACKEND Veth10 Net Adapter 1 IESEDBINTPRE
Veth11 Net Adapter 1 IESEDBPRO
IESE_PRE_INT Veth3 Net Adapter 1 IESEWEBPRE
Veth4 Net Adapter 1 IESEAPPINT
Veth5 Net Adapter 1 IESEAPPPRE1
Veth6 Net Adapter 1 IESEAPPPRE2
Veth7 Net Adapter 1 IESEAUTOPRE
Veth8 Net Adapter 1 IESECMINTPRE
DATA-UPLINK-100m Po6
SERVICE-CONSOLE Veth1 vmk0 Module 4
Veth2 vmk1 Module 4
Veth12 vmk0 Module 5
Veth13 vmk1 Module 5
DATA-UPLINK-IESE Po4
Po5
Eth4/1 vmnic0 10.100.100.63
Eth4/2 vmnic1 10.100.100.63
Eth4/3 vmnic2 10.100.100.63
Eth4/4 vmnic3 10.100.100.63
Eth5/1 vmnic0 10.100.100.64
Eth5/2 vmnic1 10.100.100.64
Eth5/3 vmnic2 10.100.100.64
Eth5/4 vmnic3 10.100.100.64
Solved! Go to Solution.
01-09-2012 09:54 AM
Hello Oscar,
Please open a TAC service request to further troubleshoot this issue.
Following additional info would help here
Is this new configuration ?
Are the uplinks connecting to same physical switch or different switches ?
Is this issue only specific to this port channel ?
Padma
01-09-2012 02:38 AM
Hello Oscar,
This generally happens if a command on the port profile failed to be appllied.
Can you send the output of the "show logging logfile | grep INTERFACE_CMD_FAILURE" and "show accounting log"?
You can look for the failed commands on the quarantined interfaces.
Thanks,
Chetan
01-09-2012 02:42 AM
Hi Chetan,
The first one shows nothing,
SW-OUT-N1KV# show logging logfile | grep INTERFACE_CMD_FAILURE
SW-OUT-N1KV#
And the second one shows the following,
Mon Jan 9 12:01:38 2012:update:192.168.237.49@pts/3:admin:configure terminal ; interface port-channel3 (SUCCESS)
Mon Jan 9 12:01:41 2012:update:192.168.237.49@pts/3:admin:configure terminal ; interface port-channel3 ; no shutdown (REDIRECT)
Mon Jan 9 12:01:41 2012:update:
192.168.237.49@pts/3:admin:configure
terminal ; interface port-channel3 (SUCCESS)
Mon Jan 9 12:01:41 2012:update:
192.168.237.49@pts/3:admin:configure
terminal ; interface port-channel3 ; no shutdown (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; switchport mode trunk (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; duplex auto (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; speed auto (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; switchport trunk native vlan 666 (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.873:admin:configure terminal ; interface port-channel3 ; switchport trunk allowed vlan 2, 43-44 (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.873:admin:configure terminal ; cdp enable (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.883:admin:configure terminal ; interface port-channel3 (SUCCESS)
Mon Jan 9 12:01:42 2012:update:ppm.883:admin:configure terminal ; interface port-channel3 ; shutdown (FAILURE)
Thanks,
01-09-2012 09:34 AM
Debugging port-profile errors, and trying agair, I get the following output:
SW-OUT-N1KV(config-port-prof)# no shut
2012 Jan 9 14:43:02.102017 port-profile: (ERR) ppm_fq_cmd_intf_range_get(1753): Failed to find intf tlv ^Yex^Yconf^Yport-profile DATA-UPLINK-IESE-SA^Yno shutdown error: Command Parsing Failed
2012 Jan 9 14:43:02.102577 port-profile: (ERR) ppm_profile_fsm_session_get(71): RID before session start request is 1
2012 Jan 9 14:43:02.106573 port-profile: (ERR) ppm_req_dbmgr_send(426): Changing session status: 0x0
2012 Jan 9 14:43:02.443092 port-profile: (ERR) ppm_vppm_profile_acfg_gen(353): VPPM Profile Level -1 acfg generation failed Error: no such pss key (0x40480003)
2012 Jan 9 14:43:03.048365 port-profile: (ERR) ppm_pending_req_queue_check(1035): msg_ref_p->state == 0xbba6e3cc
SW-OUT-N1KV(config-port-prof)# 2012 Jan 9 14:43:03.373461 port-profile: (ERR) ppm_pending_req_queue_check(1035): msg_ref_p->state == 0xbba6e3cc
SW-OUT-N1KV(config-port-prof)# channel-group auto mode active
2012 Jan 9 15:37:32.338832 port-profile: (ERR) ppm_fq_cmd_intf_range_get(1753): Failed to find intf tlv ^Yex^Yconf^Yport-profile type ethernet DATA-UPLINK-IESE-SA^Ychannel-group auto mode active error: Command Parsing Failed
2012 Jan 9 15:37:32.339399 port-profile: (ERR) ppm_profile_fsm_session_get(71): RID before session start request is 1
2012 Jan 9 15:37:32.358546 port-profile: (ERR) ppm_db_port_channel_check(7068): Not a channel grp command.
2012 Jan 9 15:37:32.360130 port-profile: (ERR) ppm_req_dbmgr_send(426): Changing session status: 0x0
2012 Jan 9 15:37:32.651020 port-profile: (ERR) ppm_vppm_profile_acfg_gen(353): VPPM Profile Level -1 acfg generation failed Error: no such pss key (0x40480003)
2012 Jan 9 15:37:33.181682 port-profile: (ERR) ppm_pending_req_queue_check(1035): msg_ref_p->state == 0xbba6e3cc
2012 Jan 9 15:37:33.387996 port-profile: (ERR) ppm_find_intf_in_cache(917): No pending plan present
2012 Jan 9 15:37:34.772375 port-profile: (ERR) ppm_pending_req_queue_check(1035): msg_ref_p->state == 0xbba6e3cc
2012 Jan 9 15:37:34.774586 port-profile: (ERR) ppm_profile_fsm_session_get(71): RID before session start request is 1
SW-OUT-N1KV(config-port-prof)# 2012 Jan 9 15:37:35.102895 port-profile: (ERR) ppm_vppm_profile_acfg_gen(353): VPPM Profile Level -1 acfg generation failed Error: no such pss key (0x40480003)
2012 Jan 9 15:37:35.469793 port-profile: (ERR) ppm_config_merge_handler(773): Show run merge not performed with reason PPM show run merge is not required
2012 Jan 9 15:37:35.502211 port-profile: (ERR) ppm_cmd_desc_create(1176): command to be ignored parsed mode:/exec/configuremode: /exec/configure cmd:version 4.2(1)SV1(4) error: Command is not a port-profile command
2012 Jan 9 15:37:35.503417 port-profile: (ERR) ppm_cmd_desc_create(1265): interface command detected ^Yex^Yconf^Yinterface port-channel5, error: Interface command is given for cmd desciptor
2012 Jan 9 15:37:35.510826 port-profile: (ERR) ppm_req_dbmgr_send(426): Changing session status: 0x0
2012 Jan 9 15:37:59.800813 port-profile: (ERR) ppm_dispatcher_invoker(542): Invoker parent failed with Err[0x1400]
2012 Jan 9 15:37:59.832496 port-profile: (ERR) procjobcb_job_done(686): status: 0x1400, (null)
2012 Jan 9 15:37:59.837319 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1250): Changing session status: 0x420c007c
2012 Jan 9 15:37:59.837732 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1252): apply status: 0x1400
2012 Jan 9 15:37:59.838435 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1277): Could not open the file /dev/shm/ppm_DATA-UPLINK-IESE-SA_output.txt
2012 Jan 9 15:37:59.800813 port-profile: (ERR) ppm_dispatcher_invoker(542): Invoker parent failed with Err[0x1400]
2012 Jan 9 15:37:59.832496 port-profile: (ERR) procjobcb_job_done(686): status: 0x1400, (null)
2012 Jan 9 15:37:59.837319 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1250): Changing session status: 0x420c007c
2012 Jan 9 15:37:59.837732 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1252): apply status: 0x1400
2012 Jan 9 15:37:59.838435 port-profile: (ERR) ppm_profile_fsm_sma_apply_acc(1277): Could not open the file /dev/shm/ppm_DATA-UPLINK-IESE-SA_output.txt
2012 Jan 9 15:37:59 SW-OUT-N1KV %PORT-PROFILE-2-INTERFACE_QUARANTINED: Interface port-channel5 has been quarantined due to Cmd Failure
Any idea? I'm lost...
01-09-2012 09:54 AM
Hello Oscar,
Please open a TAC service request to further troubleshoot this issue.
Following additional info would help here
Is this new configuration ?
Are the uplinks connecting to same physical switch or different switches ?
Is this issue only specific to this port channel ?
Padma
01-10-2012 08:37 AM
I'm sorry. I clicked on "correct answer" by mistake.
The same port-profile is working fine on other interfaces on other hosts. I'm assuming there is no problem having different hosts using the same port-profile.
The uplinks are connecting to two upsteam physical stacked switches.
I have done several tests and the results are quite confusing, at least for me.
All the interfaces on host A are working fine in PortChannel using port-profile DATA-UPLINK-IESE
All the interfaces on host B are working fine in PortChannel using port-profile DATA-UPLINK-IESE-SA
Upsteam switches' configuration are equal in both Port-Channels.
SW-OUT-N1KV# sh run port-prof DATA-UPLINK-IESE
!Command: show running-config port-profile DATA-UPLINK-IESE
!Time: Tue Jan 10 18:22:51 2012
version 4.2(1)SV1(4)
port-profile type ethernet DATA-UPLINK-IESE
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 85-86,88
switchport trunk native vlan 666
speed auto
duplex auto
cdp enable
channel-group auto mode active
no shutdown
state enabled
SW-OUT-N1KV# sh run port-prof DATA-UPLINK-IESE-SA
!Command: show running-config port-profile DATA-UPLINK-IESE-SA
!Time: Tue Jan 10 18:23:03 2012
version 4.2(1)SV1(4)
port-profile type ethernet DATA-UPLINK-IESE-SA
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 85-86,88
switchport trunk native vlan 666
speed auto
duplex auto
cdp enable
channel-group auto mode active
no shutdown
state enabled
When I try to bind PNICs on host A to the port-profile DATA-UPLINK-IESE-SA, the PortChannel is quarantined.
The same, when I try to bind interfaces on host B to the port-profile DATA-UPLINK-IESE.
I have solved the issue, creating a new port-profile, copying the configuration from the above ones, and binding all the interfaces on hosts A and B to that new port-profile.
SW-OUT-N1KV# sh run port-prof DATA-UPLINK-PROV
!Command: show running-config port-profile DATA-UPLINK-PROV
!Time: Tue Jan 10 18:28:33 2012
version 4.2(1)SV1(4)
port-profile type ethernet DATA-UPLINK-PROV
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 85-86,88
switchport trunk native vlan 666
speed auto
duplex auto
channel-group auto mode active
no shutdown
state enabled
-------------------------------------------------------------------------------
Port Profile Port Adapter Owner
-------------------------------------------------------------------------------
DATA-UPLINK-PROV Po3
Po4
Eth4/1 vmnic0 10.100.100.63
Eth4/2 vmnic1 10.100.100.63
Eth4/3 vmnic2 10.100.100.63
Eth4/4 vmnic3 10.100.100.63
Eth5/1 vmnic0 10.100.100.64
Eth5/2 vmnic1 10.100.100.64
Eth5/3 vmnic2 10.100.100.64
Eth5/4 vmnic3 10.100.100.64
Do you have any idea why is it happening?
Thanks,
01-10-2012 08:59 AM
Oscar,
Mistakes do happen :-)
We need logs to better understand the behavior. If you have not done already,I would suggest you to open SR and upload the logs to better understand the behavior.
Padma
01-11-2012 01:15 AM
Yes, I am also opening a SR.
What kind of logs do you need? the "show log logfile"? or anything else more spcific?
Thanks,
01-11-2012 01:36 AM
Oscar,
To start with, we need VSM and VEM show tech bundle
VSM
show tech-support svs | no-more
VEM ( SSh into ESXi host and execute the command )
vem-support all
Padma
01-11-2012 03:52 AM
01-11-2012 05:23 AM
Oscar,
Thanks for the logs.
Once you have opened TAC SR, engineer would follow up with you on the analysis.
I will also check it out.
Padma
01-13-2012 01:23 AM
Thank you for everything Padramas,
the engineer is already asking me for some information.
I have realised that deleting the port-profile and re-creating it (with the same config) solves the problem. After that, I can bind the interfaces with the new port-profile and everything is fine. But there must be another way to remove this "state" in the port-profile without the need of deleting it completely.
Do you know any way to do so?
Thanks
01-16-2012 04:24 AM
Oscar,
Can you please send me a private message with your TAC SR number ?
Padma
06-06-2012 09:57 AM
Hello.
I've exactly the same problem in my production site.
Please, can you post the solution the TAC gave to you ? Or can you send it me in a private message ?
Thank you.
06-08-2012 03:17 AM
Hello Julien,
I could not find a TAC service request for this issue.
Please open one so that we can help you out.
Padma
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide