vPC N5k Switch Failure causes connectivity disruption

[Bogdan Nita]

Hello,

I have configured enhanced vPC on 2 n5k and B22 FEXs (vPC from 5k to B22, and vPC from B22 to blade servers).

Everything is running smoothly, except when I power off one of the 5k, the connectivity to the blade servers is lost, comes back up for a short while, loses connectivity again, and after a few minutes comes back up for good.

From the logs I can see that all the port-channels (peer-link, to the FEXs and port-channels to other switches in network) get in down state, then fizical interfaces start coming backup in fabric mode, then port-channels, see FEXs starting to get online then all the port-channels go down again and then whole thing starts again.(all of this is hapenig with one of the 5k powerd off, same thing hapens with promary and secondary vpc).

Connectivity is lost in the same way when the 5k is started again, but just once.

I am running NX OS version 5.2.1N1.3.

I have no idea what could cause this behavior.

Any help would be appreciated.

Regards,

Bogdan

[Reza Sharifi]

Hi,

When both 5Ks are up and running, can you post the output of "sh vpc"?

Also can you post your peer-link config and also your peer keep-alive config?

Or just "sh run" from both 5Ks and post.

HTH

[Bogdan Nita]

Hi Reza,

Below you can find my run-config.

version 5.2(1)N1(3)

feature fcoe

install feature-set virtualization

feature-set virtualization

logging level feature-mgr 0

hostname N5k_1

feature npiv

feature telnet

cfs eth distribute

feature udld

feature interface-vlan

feature lacp

feature vpc

feature lldp

feature vtp

feature fex

fex 107

  pinning max-links 1

  description "FEX0107"

  fcoe

fex 108

  pinning max-links 1

  description "FEX0108"

slot 1

  port 31-32 type fc

vpc domain 1

  role priority 1000

  peer-keepalive destination 1.1.1.2

  auto-recovery

vsan database

  vsan 50 name "VSAN_A"

fcdomain fcid database

interface port-channel100

  description Po Synch N5k

  switchport mode trunk

  spanning-tree port type network

  logging event port link-status

  logging event port trunk-status

  speed 10000

  vpc peer-link

interface port-channel107

  switchport mode fex-fabric

  fex associate 107

  vpc 107

interface port-channel108

  switchport mode fex-fabric

  fex associate 108

  vpc 108

interface port-channel111

  switchport mode trunk

interface vfc111

  bind interface Ethernet107/1/1

  no shutdown

vsan database

  vsan 50 interface vfc111

  vsan 50 interface fc1/31

  vsan 50 interface fc1/32

interface fc1/31

  no shutdown

interface fc1/32

  no shutdown

interface Ethernet1/1

  description Synch N5k

  switchport mode trunk

  logging event port link-status

  logging event port trunk-status

  udld aggressive

  channel-group 100 mode active

interface Ethernet1/2

  description Synch N5k

  switchport mode trunk

  logging event port link-status

  logging event port trunk-status

  udld aggressive

  channel-group 100 mode active

interface Ethernet1/3

  description Synch N5k

  switchport mode trunk

  logging event port link-status

  logging event port trunk-status

  udld aggressive

  channel-group 100 mode active

interface Ethernet1/4

  description Synch N5k

  switchport mode trunk

  logging event port link-status

  logging event port trunk-status

  udld aggressive

  channel-group 100 mode active

interface Ethernet1/5

  switchport mode fex-fabric

  fex associate 107

  channel-group 107

interface Ethernet1/6

  switchport mode fex-fabric

  fex associate 107

  channel-group 107

interface Ethernet1/7

  switchport mode fex-fabric

  fex associate 108

  channel-group 108

interface Ethernet1/8

  switchport mode fex-fabric

  fex associate 108

  channel-group 108

interface Ethernet107/1/1

  switchport mode trunk

  channel-group 111 mode active

interface Ethernet108/1/1

  switchport mode trunk

  channel-group 111 mode active

###########################

version 5.2(1)N1(3)

feature fcoe

install feature-set virtualization

feature-set virtualization

logging level feature-mgr 0

hostname N5k_2

feature npiv

feature telnet

cfs eth distribute

feature udld

feature interface-vlan

feature lacp

feature vpc

feature lldp

feature vtp

feature fex

fex 107

  pinning max-links 1

  description "FEX0107"

fex 108

  pinning max-links 1

  description "FEX0108"

  fcoe

slot 1

  port 31-32 type fc

vpc domain 1

  role priority 1000

  peer-keepalive destination 1.1.1.1

  auto-recovery

vsan database

  vsan 51 name "VSAN_B"

fcdomain fcid database

interface port-channel100

  description Po Synch N5k

  switchport mode trunk

  spanning-tree port type network

  logging event port link-status

  logging event port trunk-status

  speed 10000

  vpc peer-link

interface port-channel107

  switchport mode fex-fabric

  fex associate 107

  vpc 107

interface port-channel108

  switchport mode fex-fabric

  fex associate 108

  vpc 108

interface port-channel111

  switchport mode trunk

interface vfc111

  bind interface Ethernet108/1/1

  no shutdown

vsan database

  vsan 51 interface vfc111

  vsan 51 interface fc1/31

  vsan 51 interface fc1/32

interface fc1/31

  no shutdown

interface fc1/32

  no shutdown

interface Ethernet1/1

  description Synch N5k

  switchport mode trunk

  logging event port link-status

  logging event port trunk-status

  udld aggressive

  channel-group 100 mode active

interface Ethernet1/2

  description Synch N5k

  switchport mode trunk

  logging event port link-status

  logging event port trunk-status

  udld aggressive

  channel-group 100 mode active

interface Ethernet1/3

  description Synch N5k

  switchport mode trunk

  logging event port link-status

  logging event port trunk-status

  udld aggressive

  channel-group 100 mode active

interface Ethernet1/4

  description Synch N5k

  switchport mode trunk

  logging event port link-status

  logging event port trunk-status

  udld aggressive

  channel-group 100 mode active

interface Ethernet1/5

  switchport mode fex-fabric

  fex associate 107

  channel-group 107

interface Ethernet1/6

  switchport mode fex-fabric

  fex associate 107

  channel-group 107

interface Ethernet1/7

  switchport mode fex-fabric

  fex associate 108

  channel-group 108

interface Ethernet1/8

  switchport mode fex-fabric

  fex associate 108

  channel-group 108

interface Ethernet107/1/1

  switchport mode trunk

  channel-group 111 mode active

interface Ethernet108/1/1

  switchport mode trunk

  channel-group 111 mode active

[Reza Sharifi]

I see, you have "role priority 1000" to both 5Ks.  Usually you assign a lower role priority for the switch you want to be the primary vPC.  What switch is the primary?

What is the output of "sh vpc"?

[Ridma Ransara Jayaweera]

Hi Bogdan

How did you check the server connectivity?? ping??

Did you check the spanning-tree state of these links???

which one is the spanning-tree root???

does same thing happen when you shutdown the spanning tree non root bridge??

if it seems issue with spaning-tee I suggest you to use use peer-switch command under vpc domain

Rid

[Bogdan Nita]

Hi Ridma,

I also did thought about STP, but I didn't find any STP log messages, and also it would explain connectivity disruption for upstream links, but it does not explain FEXs going offline.

I will try your suggestion, but my access is limited and it might take awhile until I get the chance.

Bogdan