Solved: Nexus 1000V - Port Suspended

DanielSheppard · ‎03-09-2015

Good Afternoon,

I am starting to work with the Nexus 1000V switch, however I am running into a little bump and would like some assistance if possible.

I have a dell R720 with:

2x Broadcom NetXtreme 1G

2xIntel Ethernet 10G 2P X540-T

I am included the models because I think that might somehow be part of the problem. I am getting the following interface history and any help would be appreciated in deciphering this log:

show system internal ethpm event-history interface Eth4/3

>>>>FSM: <Ethernet4/3> has 46 logged transitions<<<<<

1) FSM:<Ethernet4/3> Transition at 330000 usecs after Mon Mar 9 08:31:12 2015
Previous state: [ETH_PORT_FSM_ST_NOT_INIT]
Triggered event: [ETH_PORT_FSM_EV_MODULE_INIT_DONE]
Next state: [ETH_PORT_FSM_ST_INIT_EVAL]

2) FSM:<Ethernet4/3> Transition at 330000 usecs after Mon Mar 9 08:31:12 2015
Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
Triggered event: [ETH_PORT_FSM_EV_IE_PORT_INIT]
Next state: [ETH_PORT_FSM_ST_SPAN_EVAL]

3) FSM:<Ethernet4/3> Transition at 330000 usecs after Mon Mar 9 08:31:12 2015
Previous state: [ETH_PORT_FSM_ST_SPAN_EVAL]
Triggered event: [ETH_PORT_FSM_EV_NON_SPAN_DEST]
Next state: [ETH_PORT_FSM_ST_WAIT_PRE_CFG]

4) Event:ESQ_START length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq Type:SERIAL

5) Event:ESQ_REQ length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq:determine steps to skip

6) Event:ESQ_REQ length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x0014600A

7) Event:ESQ_RSP length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x0014600C

8) Event:ESQ_REQ length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_VSIM(716), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

9) Event:ESQ_REQ length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_DHCP_SNOOP(360), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

10) Event:ESQ_REQ length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_PVLAN(180), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

11) Event:ESQ_REQ length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_FWM(602), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x0014601B

12) Event:ESQ_RSP length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_FWM(602), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x00146020

13) Event:ESQ_REQ length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_ETH_PORT_SEC(191), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x00146026

14) Event:ESQ_RSP length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_ETH_PORT_SEC(191), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x00146028

15) Event:ESQ_REQ length:38, at 330000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_ACLMGR(351), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x0014602E

16) Event:ESQ_RSP length:38, at 340000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_ACLMGR(351), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x00146030

17) Event:ESQ_REQ length:38, at 340000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_IPQOS_MGR(377), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x00146036

18) Event:ESQ_RSP length:38, at 340000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_IPQOS_MGR(377), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
RRtoken:0x00146038

19) Event:ESQ_REQ length:38, at 340000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

20) Event:ESQ_REQ length:38, at 340000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq:Pacer Check

21) Event:ESQ_REQ length:38, at 340000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_LCP_SET_PORT_CFG(61443)
RRtoken:0x0014603F

22) Event:ESQ_RSP length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_LCP_SET_PORT_CFG(61443)
RRtoken:0x00146041

23) FSM:<Ethernet4/3> Transition at 350000 usecs after Mon Mar 9 08:31:12 2015
Previous state: [ETH_PORT_FSM_ST_WAIT_PRE_CFG]
Triggered event: [ETH_PORT_FSM_EV_PRE_CFG_DONE]
Next state: [ETH_PORT_FSM_ST_LINK_INIT]

24) FSM:<Ethernet4/3> Transition at 350000 usecs after Mon Mar 9 08:31:12 2015
Previous state: [ETH_PORT_FSM_ST_LINK_INIT]
Triggered event: [ETH_PORT_FSM_EV_LINK_UP]
Next state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]

25) Event:ESQ_START length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq Type:SERIAL

26) Event:ESQ_REQ length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq:determine steps to skip

27) Event:ESQ_REQ length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq:check for issu

28) Event:ESQ_REQ length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_PHY_LINK_STATE_CHANGE(62470)

29) Event:ESQ_REQ length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

30) Event:ESQ_RSP length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

31) Event:ESQ_REQ length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_UDLD(216), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)

32) Event:ESQ_REQ length:38, at 350000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
RRtoken:0x001460A0

33) Event:ESQ_RSP length:38, at 370000 usecs after Mon Mar 9 08:31:12 2015
Instance:620806272, Seq Id:0x1, Ret:port not compatible
[E_MTS_RX] Src:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
RRtoken:0x001460A0

34) FSM:<Ethernet4/3> Transition at 370000 usecs after Mon Mar 9 08:31:12 2015
Previous state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]
Triggered event: [ETH_PORT_FSM_EV_SUSPEND_PORT]
Next state: [ETH_PORT_FSM_ST_PORT_SUSPENDED]

35) Event: E_DEBUG length:42, at 370000 usecs after Mon Mar 9 08:33:30 2015
[REINIT] Src:MTS_SAP_ETH_PORT_CHANNEL_MGR(378) Reinit-Type:0

36) FSM:<Ethernet4/3> Transition at 370000 usecs after Mon Mar 9 08:33:30 2015
Previous state: [ETH_PORT_FSM_ST_PORT_SUSPENDED]
Triggered event: [ETH_PORT_FSM_EV_EXTERNAL_REINIT_NO_FLAP_REQ]
Next state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]

37) Event:ESQ_START length:38, at 370000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq Type:SERIAL

38) Event:ESQ_REQ length:38, at 370000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq:determine steps to skip

39) Event:ESQ_REQ length:38, at 370000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
Seq:check for issu

40) Event:ESQ_REQ length:38, at 370000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_PHY_LINK_STATE_CHANGE(62470)

41) Event:ESQ_REQ length:38, at 370000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

42) Event:ESQ_RSP length:38, at 380000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

43) Event:ESQ_REQ length:38, at 380000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_UDLD(216), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)

44) Event:ESQ_REQ length:38, at 380000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
RRtoken:0x0014648A

45) Event:ESQ_RSP length:38, at 380000 usecs after Mon Mar 9 08:33:30 2015
Instance:620806272, Seq Id:0x1, Ret:port not compatible
[E_MTS_RX] Src:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
RRtoken:0x0014648A

46) FSM:<Ethernet4/3> Transition at 380000 usecs after Mon Mar 9 08:33:30 2015
Previous state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]
Triggered event: [ETH_PORT_FSM_EV_SUSPEND_PORT]
Next state: [ETH_PORT_FSM_ST_PORT_SUSPENDED]

Curr state: [ETH_PORT_FSM_ST_PORT_SUSPENDED]

mizhang · ‎03-11-2015

Hi Daniel -

Thanks for the updates.

For port-channel, all the ports should have same NIC speed, just like regular physical switch port-channel config. requirement.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_4/interface/configuration/guide/n1000v_if/n1000v_if_5portchannel.html#wp1270393

"

Each port can be in only one port channel. All the ports in a port channel must be compatible; they must use the same speed and duplex mode

"

If you have multiple uplink port-channels (1 2G port-channel and 1 20G port-channel) from the same VEM/host, you need to avoid overlapping vlans issue.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_4/port_profile/configuration/guide/n1000v_port_profile/n1000v_portprof_4system.html

"

In an installation where multiple Ethernet port profiles are active on the same VEM, it is recommended that they do not carry the same VLAN(s). The allowed VLAN list should be mutually exclusive. Overlapping VLANs can be configured but may cause duplicate packets to be received by virtual machines in the network.

"

Regards,

Michael

View solution in original post

mizhang · ‎03-10-2015

hi -

A few more questions -

x- what version of N1Kv? -> [show ver ]

x- status of the interface e4/3 -> [show interface e4/3 ] and [show run int e4/3 expand ]

x- [show logg log | in suspend ]

You can try the following to see if it could fix the issue:

x- create another temp uplink port-profile, and migrate the vminc(i.e. e4/3) to this temp port-profile;

x- if the port comes up fine, then migrate it back to the original port-profile.

Hope it helps.

Michael

DanielSheppard · ‎03-10-2015

I was thinking it might be due to 2 of the NICs being 10G and 2 being 1G, but it sounds like you don't think this is the case.

I did try creating two different port profiles and having the 2 10G on one profile and the 2 1G on another. They both come up in that case (but due to not being 100% familiar with the Nexus line yet, I didn't know you couldn't have all VLANs out both, so it will take some re-engineering if I want to do that). I moved it back to the original profile and it went into a suspended state again.

Can you have Port-Channels with different speed NICs?

show ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
loader: version unavailable [last: loader version not available]
kickstart: version 5.2(1)SM3(1.1)
system: version 5.2(1)SM3(1.1)
kickstart image file is: bootflash:///n1000vh-dk9-kickstart.5.2.1.SM3.1.1.bin
kickstart compile time: 12/4/2014 10:00:00 [12/04/2014 19:07:52]
system image file is: bootflash:///n1000vh-dk9.5.2.1.SM3.1.1.bin
system compile time: 12/4/2014 10:00:00 [12/04/2014 20:49:28]

Hardware
cisco Nexus 1000V Chassis ("Virtual Supervisor Module")
Intel(R) Xeon(R) CPU with 4135820 kB of memory.
Processor Board ID T155D358504

Device name: SW-06
bootflash: 3122988 kB

Kernel uptime is 3 day(s), 23 hour(s), 12 minute(s), 1 second(s)

plugin
Core Plugin, Ethernet Plugin, Virtualization Plugin

Ethernet4/3 is down (suspended)
Hardware: Ethernet, address: c81f.66b9.df80 (bia c81f.66b9.df80)
Port-Profile is ACSNET-UPLINK
MTU 1500 bytes
Encapsulation ARPA
Port mode is trunk
auto-duplex, 1000 Mb/s
5 minute input rate 361232 bits/second, 75 packets/second
5 minute output rate 32 bits/second, 0 packets/second
Rx
5638365 Input Packets 2181 Unicast Packets
5203238 Multicast Packets 432952 Broadcast Packets
3244083521 Bytes
Tx
2363 Output Packets 378 Unicast Packets
1363 Multicast Packets 623 Broadcast Packets 708 Flood Packets
542998 Bytes
112389 Input Packet Drops 0 Output Packet Drops

interface Ethernet4/3
switchport mode trunk
switchport trunk allowed vlan 1,5,11-14,100,110,120-122,125,130
switchport trunk allowed vlan add 140-145,200
channel-group auto mode on mac-pinning
no shutdown

2015 Mar 9 12:20:27 SW-06 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: port-channel7: port-channel7 is suspended
2015 Mar 9 12:20:27 SW-06 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: port-channel7: port-channel7 is suspended

mizhang · ‎03-11-2015

Hi Daniel -

Thanks for the updates.

For port-channel, all the ports should have same NIC speed, just like regular physical switch port-channel config. requirement.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_4/interface/configuration/guide/n1000v_if/n1000v_if_5portchannel.html#wp1270393

"

Each port can be in only one port channel. All the ports in a port channel must be compatible; they must use the same speed and duplex mode

"

If you have multiple uplink port-channels (1 2G port-channel and 1 20G port-channel) from the same VEM/host, you need to avoid overlapping vlans issue.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_4/port_profile/configuration/guide/n1000v_port_profile/n1000v_portprof_4system.html

"

In an installation where multiple Ethernet port profiles are active on the same VEM, it is recommended that they do not carry the same VLAN(s). The allowed VLAN list should be mutually exclusive. Overlapping VLANs can be configured but may cause duplicate packets to be received by virtual machines in the network.

"

Regards,

Michael