Nexus 3k All servers ports flapping

[Latchum Naidu]

Hi All,

I was doing one Nexus 3k switch replacement and following abnormalities what I observed....

1. After everything done like licensing and config push I enabled the int eht1/25 which is part of Po1 (Po1 have members eth1/25 & 1/26), I found that 1/25 went into suspended state and when enabled 1/26 then both came up and along with PO1.... Why was this case?

2. After confirmed all PO's (uplinks) up and stable I enabled remaining all server connected ports and found all are getting flap (only server ports) When I enabled the all BGP neighbors on the switch then found everything stable including server ports.... why was this case?

Below are the logs which were captured during replacement.

ch3-x3hl-cis-2-05# sh log last 100
2014 Jan 11 03:19:21.244 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/9 is down (Link failure)
2014 Jan 11 03:19:21.554 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/13 is down (Link failure)
2014 Jan 11 03:19:21.914 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode access
2014 Jan 11 03:19:22.030 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)
2014 Jan 11 03:19:22.059 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/2 is up in mode access
2014 Jan 11 03:19:23.084 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/7 is down (Link failure)
2014 Jan 11 03:19:24.504 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode access
2014 Jan 11 03:19:24.541 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/51/4 is up in mode access
2014 Jan 11 03:19:24.552 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)
2014 Jan 11 03:19:24.622 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/9 is up in mode access
2014 Jan 11 03:19:24.712 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/52/3 is down (Link failure)
2014 Jan 11 03:19:25.457 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/52/4 is down (Link failure)
2014 Jan 11 03:19:26.097 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/13 is up in mode access
2014 Jan 11 03:19:26.147 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode access
2014 Jan 11 03:19:26.372 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/7 is up in mode access
2014 Jan 11 03:19:26.827 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/3 is down (Link failure)
2014 Jan 11 03:19:27.839 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode access
2014 Jan 11 03:19:27.854 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/3 is up in mode access
2014 Jan 11 03:19:27.856 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/4 is up in mode access
2014 Jan 11 03:19:29.292 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/3 is up in mode access
2014 Jan 11 03:19:33.451 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure)
2014 Jan 11 03:19:34.410 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/11 is down (Link failure)
2014 Jan 11 03:19:34.514 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/14 is down (Link failure)
2014 Jan 11 03:19:36.177 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/8 is down (Link failure)
2014 Jan 11 03:19:36.223 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/4 is up in mode access
2014 Jan 11 03:19:37.166 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/11 is up in mode access
2014 Jan 11 03:19:37.171 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/14 is up in mode access
2014 Jan 11 03:19:38.641 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/8 is up in mode access
2014 Jan 11 03:20:23.932 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/52/2 is down (Link failure)
2014 Jan 11 03:20:25.211 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/5 is down (Link failure)
2014 Jan 11 03:20:26.142 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/9 is down (Link failure)
2014 Jan 11 03:20:26.267 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/2 is up in mode access
2014 Jan 11 03:20:27.104 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode access
2014 Jan 11 03:20:27.517 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/13 is down (Link failure)
2014 Jan 11 03:20:28.475 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)
2014 Jan 11 03:20:28.586 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/9 is up in mode access
2014 Jan 11 03:20:29.353 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/1 is down (Link failure)
2014 Jan 11 03:20:29.490 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/13 is up in mode access
2014 Jan 11 03:20:30.821 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)
2014 Jan 11 03:20:30.927 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/7 is down (Link failure)
2014 Jan 11 03:20:30.989 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode access
2014 Jan 11 03:20:32.627 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/3 is down (Link failure)
2014 Jan 11 03:20:32.740 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/52/3 is down (Link failure)
2014 Jan 11 03:20:32.742 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/52/4 is down (Link failure)
2014 Jan 11 03:20:32.794 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode access
2014 Jan 11 03:20:33.760 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/51/4 is down (Link failure)
2014 Jan 11 03:20:33.792 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode access
2014 Jan 11 03:20:33.800 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/7 is up in mode access
2014 Jan 11 03:20:34.133 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/3 is up in mode access
2014 Jan 11 03:20:34.142 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/4 is up in mode access
2014 Jan 11 03:20:35.110 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/3 is up in mode access
2014 Jan 11 03:20:35.119 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/51/4 is up in mode access
2014 Jan 11 03:20:35.590 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/2 is down (Link failure)
2014 Jan 11 03:20:36.935 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/6 is down (Link failure)
2014 Jan 11 03:20:37.902 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/2 is up in mode access
2014 Jan 11 03:20:38.812 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure)
2014 Jan 11 03:20:39.712 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/14 is down (Link failure)
2014 Jan 11 03:20:39.795 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/6 is up in mode access
2014 Jan 11 03:20:40.153 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/8 is down (Link failure)
2014 Jan 11 03:20:40.243 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/11 is down (Link failure)
2014 Jan 11 03:20:41.224 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/4 is up in mode access
2014 Jan 11 03:20:42.413 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/52/2 is down (Link failure)
2014 Jan 11 03:20:42.441 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/14 is up in mode access
2014 Jan 11 03:20:42.700 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/5 is down (Link failure)
2014 Jan 11 03:20:42.860 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/8 is up in mode access
2014 Jan 11 03:20:43.721 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/11 is up in mode access
2014 Jan 11 03:20:43.734 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/2 is up in mode access
2014 Jan 11 03:20:44.076 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/9 is down (Link failure)
2014 Jan 11 03:20:45.048 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/13 is down (Link failure)
2014 Jan 11 03:20:45.178 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode access
2014 Jan 11 03:20:46.089 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)
2014 Jan 11 03:20:46.227 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/9 is up in mode access
2014 Jan 11 03:20:47.136 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/1 is down (Link failure)
2014 Jan 11 03:20:47.176 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/13 is up in mode access
2014 Jan 11 03:20:48.071 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)
2014 Jan 11 03:20:48.212 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode access
2014 Jan 11 03:20:49.007 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/7 is down (Link failure)
2014 Jan 11 03:20:49.138 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode access
2014 Jan 11 03:20:49.912 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/3 is down (Link failure)
2014 Jan 11 03:20:50.020 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/52/3 is down (Link failure)
2014 Jan 11 03:20:50.067 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode access
2014 Jan 11 03:20:50.915 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/52/4 is down (Link failure)
2014 Jan 11 03:20:51.040 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/7 is up in mode access
2014 Jan 11 03:20:52.331 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/3 is up in mode access
2014 Jan 11 03:20:52.336 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/3 is up in mode access
2014 Jan 11 03:20:53.323 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/2 is down (Link failure)
2014 Jan 11 03:20:53.465 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/52/4 is up in mode access
2014 Jan 11 03:20:54.677 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/6 is down (Link failure)
2014 Jan 11 03:20:55.639 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/2 is up in mode access
2014 Jan 11 03:20:57.133 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure)
2014 Jan 11 03:20:57.328 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/14 is down (Link failure)
2014 Jan 11 03:20:57.557 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/6 is up in mode access
2014 Jan 11 03:20:59.081 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/8 is down (Link failure)
2014 Jan 11 03:20:59.187 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/11 is down (Link failure)
2014 Jan 11 03:21:00.764 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/4 is up in mode access
2014 Jan 11 03:21:00.777 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/14 is up in mode access
2014 Jan 11 03:21:01.729 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/8 is up in mode access
2014 Jan 11 03:21:02.266 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/11 is up in mode access
2014 Jan 11 03:21:10.578 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/51/4 is down (Link failure)
2014 Jan 11 03:21:12.407 ch3-x3hl-cis-2-05 %ETHPORT-5-IF_UP: Interface Ethernet1/51/4 is up in mode access
ch3-x3hl-cis-2-05#
ch3-x3hl-cis-2-05# sh clock
03:49:44.297 UTC Sat Jan 11 2014
ch3-x3hl-cis-2-05#

Regards,
Naidu.

[Gpetroski1]

Hello, I have the same problem on 2 ports only, but worth checking this bug. 

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCua85354/?rfs=iqvred