Re: Interface reset causes Port-channel to flap

Saleem Roumaldaro · ‎02-21-2012

interface resets causes Port channel to flap nexus 5010

Hi all,

I am having the issues explained below , I did not find any bug related to the below error messages , your help is appreciated.

Here is an example of what each switch logs when a server drops offline. Sample logs taken between 5:32am and 5:35am on Feb 20. This particular one was having problems all weekend. Switch #1 encountered over 2000 interface resets. The corresponding VPC port on Switch #2 only had 13 resets.

NEXUS SWITCH #1

2012 Feb 20 05:32:09 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from Ethernet1/10 to none

2012 Feb 20 05:32:09 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/10 is down

2012 Feb 20 05:32:09 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: port-channel10 is down

2012 Feb 20 05:32:09 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:32:09 q91-sw01-5010 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)

2012 Feb 20 05:32:09 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-SPEED: Interface Ethernet1/10, operational speed changed to 10 Gbps

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/10, operational duplex mode changed to

Full

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/10, operational Receive Flow

Control state changed to off

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/10, operational Transmit Flow

Control state changed to off

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-SPEED: Interface port-channel10, operational speed changed to 10 Gbps

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface port-channel10, operational duplex mode changed

to Full

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel10, operational Receive Flo

w Control state changed to off

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel10, operational Transmit Fl

ow Control state changed to off

2012 Feb 20 05:32:20 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/10 is up

2012 Feb 20 05:32:20 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from none to Ethernet1/10

2012 Feb 20 05:32:20 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode trunk

2012 Feb 20 05:32:20 q91-sw01-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:32:20 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface port-channel10 is up in mode trunk

2012 Feb 20 05:33:22 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from Ethernet1/10 to none

2012 Feb 20 05:33:22 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/10 is down

2012 Feb 20 05:33:22 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: port-channel10 is down

2012 Feb 20 05:33:22 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:33:22 q91-sw01-5010 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)

2012 Feb 20 05:33:22 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-SPEED: Interface Ethernet1/10, operational speed changed to 10 Gbps

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/10, operational duplex mode changed to

Full

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/10, operational Receive Flow

Control state changed to off

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/10, operational Transmit Flow

Control state changed to off

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-SPEED: Interface port-channel10, operational speed changed to 10 Gbps

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface port-channel10, operational duplex mode changed

to Full

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel10, operational Receive Flo

w Control state changed to off

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel10, operational Transmit Fl

ow Control state changed to off

2012 Feb 20 05:33:33 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/10 is up

2012 Feb 20 05:33:33 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from none to Ethernet1/10

2012 Feb 20 05:33:33 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode trunk

2012 Feb 20 05:33:33 q91-sw01-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:33:33 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface port-channel10 is up in mode trunk

2012 Feb 20 05:34:35 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from Ethernet1/10 to none

2012 Feb 20 05:34:35 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/10 is down

2012 Feb 20 05:34:35 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: port-channel10 is down

2012 Feb 20 05:34:35 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:34:35 q91-sw01-5010 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)

2012 Feb 20 05:34:35 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-SPEED: Interface Ethernet1/10, operational speed changed to 10 Gbps

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/10, operational duplex mode changed to

Full

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/10, operational Receive Flow

Control state changed to off

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/10, operational Transmit Flow

Control state changed to off

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-SPEED: Interface port-channel10, operational speed changed to 10 Gbps

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface port-channel10, operational duplex mode changed

to Full

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel10, operational Receive Flo

w Control state changed to off

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel10, operational Transmit Fl

ow Control state changed to off

2012 Feb 20 05:34:46 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/10 is up

2012 Feb 20 05:34:46 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from none to Ethernet1/10

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode trunk

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface port-channel10 is up in mode trunk

2012 Feb 20 05:35:48 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from Ethernet1/10 to none

NEXUS SWITCH #2

2012 Feb 20 05:32:20 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 999 on Interface port-channel10 are rem

oved from suspended state.

2012 Feb 20 05:32:20 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:33:33 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 999 on Interface port-channel10 are rem

oved from suspended state.

2012 Feb 20 05:33:33 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:34:46 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 999 on Interface port-channel10 are rem

oved from suspended state.

2012 Feb 20 05:34:46 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:35:59 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 999 on Interface port-channel10 are rem

oved from suspended state.

2012 Feb 20 05:35:59 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

dchhaya · ‎02-21-2012

Hello Saleem.

This looks like a physical layer problem

2012 Feb 20 05:34:35 q91-sw01-5010 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)

can you check the sfp on both sides adn the cable. can you also make sure if the server drivers are up to date. Can you connect the server to another port on the nexus and see if the same issue happens.

Saleem Roumaldaro · ‎02-21-2012

Thanks , I will test it and will let you know.

Saleem Roumaldaro

Service Consultant

Softchoice Corporation

Direct:905 847 6800 ext. 5334

Toll free: 888 436 5555

Fax: 905 847 6584

www.unislumin.com<../../../../../srei/Documents/2011/Edison/www.unislumin.com>

Leo Laohoo · ‎02-21-2012

Test using TDR.

Command:

1. test cable tdr interface ;

2. Wait for about 60 seconds (because it's a Nexus); and

3. sh cable tdr interface ; and

4. Please post the result.

Lucien Avramov · ‎02-21-2012

Link failure is most likely as it states a cable / SFP issue on the

remote side. (SFP failure on the local switch would tell you link not

connected or sfp not inserted).