Solved: Loop guard inconsistent

aok · ‎10-22-2018

Hello

We have a pair of Nexus 9363 switches connected to a Dell 5548 switch stack and a pair of Meraki MX100s connected to the Dells. The firmware on the MX100s was upgraded yesterday and after the reboot they are not able to reach their gateway. The SVI for the vlan resides on the Nexus core pair. We see this in the Nexus logs:

2018 Oct 21 23:16:47 VCR1C1R4CS1 %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel102 on VLAN0085.

Here is the vlan's configuration and spanning-tree output:

VCR1C1R4CS1# sh spanning-tree vlan 85

VLAN0085
Spanning tree enabled protocol rstp
Root ID Priority 36949
Address 0078.8810.52af
This bridge is the root
Hello Time 4 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 36949 (priority 36864 sys-id-ext 85)
Address 0078.8810.52af
Hello Time 4 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po7 Desg FWD 200 128.4102 (vPC) P2p
Po8 Desg FWD 200 128.4103 (vPC) P2p
Po9 Desg FWD 200 128.4104 (vPC) P2p
Po10 Desg FWD 200 128.4105 (vPC) P2p
Po11 Desg FWD 200 128.4106 (vPC) P2p
Po12 Desg FWD 200 128.4107 (vPC) P2p
Po23 Desg FWD 200 128.4118 (vPC) P2p
Po49 Desg FWD 250 128.4144 (vPC peer-link) Network P2p
Po101 Desg FWD 200 128.4196 (vPC) P2p
Po102 Desg BKN*200 128.4197 (vPC) P2p *LOOP_Inc
Po103 Desg FWD 200 128.4198 (vPC) P2p
Po104 Desg FWD 200 128.4199 (vPC) P2p
Po105 Desg FWD 200 128.4200 (vPC) P2p
Po106 Desg FWD 200 128.4201 (vPC) P2p
Po109 Desg FWD 200 128.4204 (vPC) P2p
Po110 Desg FWD 200 128.4205 (vPC) P2p
Po200 Desg FWD 200 128.4295 (vPC) P2p
Po201 Desg FWD 200 128.4296 (vPC) P2p
Po989 Desg FWD 200 128.5084 (vPC) P2p

___

VCR1C1R4CS1# sh run int vlan 85

!Command: show running-config interface Vlan85
!Time: Mon Oct 22 18:44:18 2018

version 7.0(3)I2(3)

interface Vlan85
description Meraki Handoff
no shutdown
ip address 172.19.11.44/29
hsrp version 2
hsrp 85
authentication md5 key-string 85
preempt delay minimum 30
priority 110
timers 1 3
ip 172.19.11.41

___

The port-channel from the Nexus to Dell's is po102 and consists of ports E1/3 and E/4. Both physical ports and the port-channel are up and passing traffic, but vlan 85 is no longer being passed through it.

___

The Dell switches are connected to the Nexus switches on ports Te1/0/1, Te1/0/2, Te2/0/1 & Te2/0/2 bundled into Po30. Ports gi1/0/19, gi1/0/20, gi2/0/19 & gi2/0/20 are all access ports in vlan 85 connecting to the two Meraki MX100s.

___

My question is....how do we get vlan 85 working on the link between the Nexus and Dell switches without impacting other traffic?

Please let me know what additional information you need.

Thanks

A

aok · ‎11-19-2018

Removing portfast from the access ports on the Dell switch resolved the issue.

View solution in original post

Reza Sharifi · ‎10-22-2018

Hi,

So, if you only have one vlan between the Nexus and dell (85) you can configure it as an access port. If you need to send multiple vlans between Nexus and Dell than Portchannle 102 needs to be configured as a trunk. It is usually best practice to configure the connection between switches as a trunk. This way if you need to add another vlan to the trunk, there will be no downtime. On the other hand, if the connection is an access port, changing it to trunk with cause a short outage.

HTH

nazimkha · ‎10-22-2018

Here is the info required to understand the issue :

1. show spanning-tree vlan 85 from both nexus
2. show vpc consistency from both switches
3. running config of both nexus
4.sh ip arp vlan 85 both nexus
5. Topology diagram

aok · ‎10-22-2018

Hi Nazimkha

The second command didn't work but here is the output for 1 & 4. The devices are named VCR1C1R4CS1 & VCR1C1R4CS2. Will scrub the running config and post along with a topology diagram.

Thanks

A

VCR1C1R4CS1# sh spanning-tree vlan 85

VLAN0085
Spanning tree enabled protocol rstp
Root ID Priority 36949
Address 0078.8810.52af
This bridge is the root
Hello Time 4 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 36949 (priority 36864 sys-id-ext 85)
Address 0078.8810.52af
Hello Time 4 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po7 Desg FWD 200 128.4102 (vPC) P2p
Po8 Desg FWD 200 128.4103 (vPC) P2p
Po9 Desg FWD 200 128.4104 (vPC) P2p
Po10 Desg FWD 200 128.4105 (vPC) P2p
Po11 Desg FWD 200 128.4106 (vPC) P2p
Po12 Desg FWD 200 128.4107 (vPC) P2p
Po23 Desg FWD 200 128.4118 (vPC) P2p
Po49 Desg FWD 250 128.4144 (vPC peer-link) Network P2p
Po101 Desg FWD 200 128.4196 (vPC) P2p
Po102 Desg BKN*200 128.4197 (vPC) P2p *LOOP_Inc
Po103 Desg FWD 200 128.4198 (vPC) P2p
Po104 Desg FWD 200 128.4199 (vPC) P2p
Po105 Desg FWD 200 128.4200 (vPC) P2p
Po106 Desg FWD 200 128.4201 (vPC) P2p
Po109 Desg FWD 200 128.4204 (vPC) P2p
Po110 Desg FWD 200 128.4205 (vPC) P2p
Po200 Desg FWD 200 128.4295 (vPC) P2p
Po201 Desg FWD 200 128.4296 (vPC) P2p
Po989 Desg FWD 200 128.5084 (vPC) P2p

VCR1C1R4CS1# sh ip arp vlan 85

Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface

IP ARP Table
Total number of entries: 3
Address Age MAC Address Interface
172.19.11.41 00:15:52 0000.0c9f.f055 Vlan85
172.19.11.45 00:15:53 00c8.8bed.e5a7 Vlan85
172.19.11.46 00:00:24 INCOMPLETE Vlan85

_____

VCR1C1R4CS2# sh spanning-tree vlan 85

VLAN0085
Spanning tree enabled protocol rstp
Root ID Priority 36949
Address 0078.8810.52af
Cost 250
Port 4144 (port-channel49)
Hello Time 4 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 41045 (priority 40960 sys-id-ext 85)
Address 00c8.8bed.e5a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po7 Desg FWD 200 128.4102 (vPC) P2p
Po9 Desg FWD 200 128.4104 (vPC) P2p
Po11 Desg FWD 200 128.4106 (vPC) P2p
Po23 Desg FWD 200 128.4118 (vPC) P2p
Po49 Root FWD 250 128.4144 (vPC peer-link) Network P2p
Po101 Desg FWD 200 128.4196 (vPC) P2p
Po102 Desg BKN*200 128.4197 (vPC) P2p *LOOP_Inc
Po103 Desg FWD 200 128.4198 (vPC) P2p
Po104 Desg FWD 200 128.4199 (vPC) P2p
Po105 Desg FWD 200 128.4200 (vPC) P2p
Po106 Desg FWD 200 128.4201 (vPC) P2p
Po109 Desg FWD 200 128.4204 (vPC) P2p
Po110 Desg FWD 200 128.4205 (vPC) P2p
Po200 Desg FWD 200 128.4295 (vPC) P2p
Po201 Desg FWD 200 128.4296 (vPC) P2p

VCR1C1R4CS2# sh ip arp vlan 85

Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface

IP ARP Table
Total number of entries: 3
Address Age MAC Address Interface
172.19.11.44 00:16:10 0078.8810.52af Vlan85
172.19.11.46 00:00:12 INCOMPLETE Vlan85
172.19.11.41 - 0000.0c9f.f055 Vlan85

aok · ‎10-22-2018

Here are the relevant pieces of the running-config, let me know if you need any other specific snippets:

VCR1C1R4CS1# sh run

version 7.0(3)I2(3)
hostname VCR1C1R4CS1
install feature-set fex
vdc VCR1C1R4CS1 id 1
allow feature-set fex
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8

feature privilege
feature telnet
feature vrrp
feature tacacs+
cfs ipv4 distribute
cfs eth distribute
feature pbr
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature vtp
feature sla sender
feature sla responder
feature sflow

vlan 1-2,4,6,8,10,12,16-18,22-23,26,28,30,32,40-47,64-66,70,84-87,90-102,104,128,211-214,282,284,286,312-315,500-502,504-505,510-512,600-601,989,999,1002-1005

spanning-tree pathcost method long
spanning-tree port type edge bpduguard default
spanning-tree port type edge bpdufilter default
spanning-tree loopguard default
spanning-tree vlan 1-3967 priority 36864
spanning-tree vlan 1-3967 hello-time 4

vpc domain 1
role priority 100
peer-keepalive destination 172.19.10.49 source 172.19.10.48
peer-gateway
auto-recovery
ip arp synchronize

vlan configuration 2

interface Vlan85
description Meraki Handoff
no shutdown
ip address 172.19.11.44/29
hsrp version 2
hsrp 85
authentication md5 key-string 85
preempt delay minimum 30
priority 110
timers 1 3
ip 172.19.11.41

interface port-channel49
description VPC Channel-group
switchport mode trunk
spanning-tree port type network
service-policy type qos input classify
vpc peer-link

interface port-channel102
description VPC to Rack 2 5548s
switchport mode trunk
vpc 102

interface Ethernet1/3
description VPC to Rack 2 5548s
switchport mode trunk
channel-group 102 mode active

interface Ethernet1/4
description VPC to Rack 2 5548s
switchport mode trunk
channel-group 102 mode active

interface Ethernet2/1
description CADC VPC Port
switchport mode trunk
spanning-tree port type network
channel-group 49 mode active

interface Ethernet2/2
description CADC VPC Port
switchport mode trunk
spanning-tree port type network
channel-group 49 mode active

_____

VCR1C1R4CS2# sh run

version 7.0(3)I2(3)
hostname VCR1C1R4CS2
install feature-set fex
vdc VCR1C1R4CS2 id 1
allow feature-set fex
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 248 maximum 248
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8

feature privilege
feature telnet
feature vrrp
feature tacacs+
cfs ipv4 distribute
cfs eth distribute
feature pbr
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature vtp
feature sla sender
feature sla responder

vlan 1-2,4,6,8,10,12,16-18,22-23,26,28,30,32,40-47,64-66,70,84-87,90-102,104,128,211-214,282,284,286,312-315,500-502,504-505,510-512,600-601,989,999,1002-1005

spanning-tree pathcost method long
spanning-tree port type edge bpduguard default
spanning-tree port type edge bpdufilter default
spanning-tree loopguard default
spanning-tree vlan 1-3967 priority 40960

track 2 ip sla 2 reachability
vpc domain 1
role priority 200
peer-keepalive destination 172.19.10.48 source 172.19.10.49
peer-gateway
auto-recovery
ip arp synchronize
vlan configuration 1

interface Vlan85
description Meraki Handoff
no shutdown
ip address 172.19.11.45/29
hsrp version 2
hsrp 85
authentication md5 key-string 85
preempt delay minimum 30
priority 110
timers 1 3
ip 172.19.11.41

interface port-channel49
description VPC Channel-group
switchport mode trunk
spanning-tree port type network
service-policy type qos input classify
vpc peer-link

interface port-channel102
description VPC to Rack 2 5548s
switchport mode trunk
vpc 102

interface Ethernet1/3
description VPC to Rack 2 5548s
switchport mode trunk
channel-group 102 mode active

interface Ethernet1/4
description VPC to Rack 2 5548s
switchport mode trunk
channel-group 102 mode active

interface Ethernet2/1
description CADC VPC Port
switchport mode trunk
spanning-tree port type network
channel-group 49 mode active

interface Ethernet2/2
description CADC VPC Port
switchport mode trunk
spanning-tree port type network
channel-group 49 mode active

aok · ‎11-01-2018

Bumping this thread, anyone have any ideas? One more thing to note is that the Nexus cores are running per-vlan rapid spanning-tree but the downstream Dell switch is running normal rapid spanning-tree.

paul driver · ‎11-01-2018

Hello

Did this only start when you upgraded the meraki's? - if so downgrade them again and see if this issue is negated?

Also you state you have the meraki's on access (edge) ports whch are switches and at the same time I see you have bpdu filtering applied on the nx-os, do you have bpdu filtering on the dell's also, if so remove it?

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

aok · ‎11-02-2018

Hi Paul

Yes it only happens after the Merakis are upgraded and rebooted. We already downgraded them as a test but it didn't make a difference. The same thing happened back in March the last time they were upgraded/rebooted. I think its just the reboot part that's causing it but can't say for sure.

I've attached the running-config of the Dell switch. Also, according to the below, bpdus are being filtered globally on it:

vcr1c1r2ds1# show spanning-tree bpdu

Global: Filtering

All of our downstream Dell switches are filtering but this is the only one (and only vlan) having issues. Anything else in the Dell switch config that might need tweaking?

Thanks for looking at this.

A

aok · ‎11-02-2018

Just noticed I didn't mention this... We tried shut/no shut on po102 and vlan 85 on the Cisco side but that didn't make any difference so ended up rebooting the Dell switch to resolve the problem. We need to upgrade the Meraki again next week so hoping to have a permanent fix in place rather than having to reboot the Dell switch to fix it every time.

aok · ‎11-05-2018

Something else to note is the spanning-tree states on the Dell switch. Ports gi1/19 & 1/20 are in Forwarding Designated state and gi2/19 & 2/20 are in Discarding Backup state, all with the same Cost value of 20000:

vcr1c1r2ds1# show spanning-tree

Name State Prio.Nbr Cost Sts Role PortFast Type
--------- -------- -------- -------- ------ ---- -------- -----------------

gi1/0/19 enabled 128.19 20000 Frw Desg No P2P (RSTP)
gi1/0/20 enabled 128.20 20000 Frw Desg No P2P (RSTP)

gi2/0/19 enabled 128.71 20000 Dscr Bkup No P2P (RSTP)
gi2/0/20 enabled 128.72 20000 Dscr Bkup No P2P (RSTP)

Also it mentions in the STP output above that portfast is not enabled, however in the configuration it says it is:

vcr1c1r2ds1# sh running-config interface gi1/0/19
interface gigabitethernet1/0/19
spanning-tree portfast
switchport access vlan 85
!
vcr1c1r2ds1# sh running-config interface gi2/0/19
interface gigabitethernet2/0/19
spanning-tree portfast
switchport access vlan 85
!

vcr1c1r2ds1# sh running-config interface gi1/0/20
interface gigabitethernet1/0/20
spanning-tree portfast
switchport access vlan 85

!

vcr1c1r2ds1# sh running-config interface gi2/0/20
interface gigabitethernet2/0/20
spanning-tree portfast
switchport access vlan 85
___

Am I missing something obvious here?

Another thought...is it possible that the order the Merakis are rebooted in might affect spanning-tree/loopguard?

We will be upgrading and rebooting the Merakis again this week so hoping to be a little closer to the root cause before then so that we don't have to reboot the Dell switch to resolve the loopguard issue.

Thanks

A

aok · ‎11-07-2018

Yet another snippet of information:

The BPDU Received count is going up on the Discarding Backup ports gi2/0/19 and gi2/0/20. These outputs were taken around 10 seconds apart:

vcr1c1r2ds1# show spanning-tree gigabitethernet 2/0/19

Port gi2/0/19 enabled

State: discarding Role: backup

Port id: 128.71 Port cost: 20000

Type: P2P (configured:Auto ) RSTP Port Fast: No (configured:Yes)

Designated bridge Priority : 32768 Address: d0:67:e5:d3:8e:de

Designated port id: 128.19 Designated path cost: 30200

Guard root: Disabled BPDU guard: Disabled

Number of transitions to forwarding state: 0

BPDU: sent 5, received 624573

vcr1c1r2ds1# show spanning-tree gigabitethernet 2/0/19

Port gi2/0/20 enabled

State: discarding Role: backup

Port id: 128.72 Port cost: 20000

Type: P2P (configured:Auto ) RSTP Port Fast: No (configured:Yes)

Designated bridge Priority : 32768 Address: d0:67:e5:d3:8e:de

Designated port id: 128.20 Designated path cost: 30200

Guard root: Disabled BPDU guard: Disabled

Number of transitions to forwarding state: 0

BPDU: sent 5, received 624586

!!!

vcr1c1r2ds1# show spanning-tree gigabitethernet 2/0/20

Port gi2/0/20 enabled

State: discarding Role: backup

Port id: 128.72 Port cost: 20000

Type: P2P (configured:Auto ) RSTP Port Fast: No (configured:Yes)

Designated bridge Priority : 32768 Address: d0:67:e5:d3:8e:de

Designated port id: 128.20 Designated path cost: 30200

Guard root: Disabled BPDU guard: Disabled

Number of transitions to forwarding state: 0

BPDU: sent 5, received 624578

vcr1c1r2ds1# show spanning-tree gigabitethernet 2/0/20

Port gi2/0/19 enabled

State: discarding Role: backup

Port id: 128.71 Port cost: 20000

Type: P2P (configured:Auto ) RSTP Port Fast: No (configured:Yes)

Designated bridge Priority : 32768 Address: d0:67:e5:d3:8e:de

Designated port id: 128.19 Designated path cost: 30200

Guard root: Disabled BPDU guard: Disabled

Number of transitions to forwarding state: 0

BPDU: sent 5, received 624584

These are the outputs on the Forwarding ports gi1/0/19 and gi1/0/20:

vcr1c1r2ds1# show spanning-tree gigabitethernet 1/0/19

Port gi1/0/19 enabled

State: forwarding Role: designated

Port id: 128.19 Port cost: 20000

Type: P2P (configured:Auto ) RSTP Port Fast: No (configured:Yes)

Designated bridge Priority : 32768 Address: d0:67:e5:d3:8e:de

Designated port id: 128.19 Designated path cost: 30200

Guard root: Disabled BPDU guard: Disabled

Number of transitions to forwarding state: 1

BPDU: sent 624645, received 3

vcr1c1r2ds1# show spanning-tree gigabitethernet 1/0/20

Port gi1/0/20 enabled

State: forwarding Role: designated

Port id: 128.20 Port cost: 20000

Type: P2P (configured:Auto ) RSTP Port Fast: No (configured:Yes)

Designated bridge Priority : 32768 Address: d0:67:e5:d3:8e:de

Designated port id: 128.20 Designated path cost: 30200

Guard root: Disabled BPDU guard: Disabled

Number of transitions to forwarding state: 1

BPDU: sent 624646, received 5

The Received count on the forwarding ports are not incrementing, however the Sent count is incrementing and the numbers match the “Received” count on the Discarding ports. So it looks like the forwarding ports are sending BPDUs and they are being received on the blocking ports. I imagine the Dell switch is just forwarding BPDUs received from the upstream Nexus and the Merakis are passing them out the other ports but can't say for sure. Is this expected behavior or does something look not quite right?

Thanks

A

paul driver · ‎11-07-2018

Hello

backup ports ! - hmm you sure you dont have a flakey cable as such it’s looping back on itself?

Looks like the port is seeing its own bpdu

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

aok · ‎11-07-2018

Hi Paul :)

Physically there aren't loops, but the MXs forward BPDUs, so if the Dell switch sends a BPDU I imagine the MX forwards it out of the other port connected to the Dell switch and it goes into backup state? Do you think removing portfast might make any difference or as that likely irrelevant?

Thanks

A

paul driver · ‎11-07-2018

Hello

i am on my phone so it’s hard to see you OP however if you have two switches connecting together DONT enable PF

Stp should not be seeing them as edge ports either

can you share the port config of both the dell and nx-os

Note- stp backup port is basically saying that port has seen it own bpdu - in stp is it being seen has a shared port ?

if so check the speed/duplex setting of the ports

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

aok · ‎11-07-2018

Hi Paul

I was referring to the access ports from the Dell switches to the Meraki firewalls having portfast enabled. The firewalls don't participate in spanning-tree, they only forward BPDUs that they receive from elsewhere. The ports between the Dell switch and Cisco Nexus are not configured as portfast.

Here's the port config for the Dell switch ports connecting to the Meraki Firewalls:

vcr1c1r2ds1# sh running-config interface gi1/0/19
interface gigabitethernet1/0/19
spanning-tree portfast
switchport access vlan 85
!
vcr1c1r2ds1# sh running-config interface gi2/0/19
interface gigabitethernet2/0/19
spanning-tree portfast
switchport access vlan 85
!

vcr1c1r2ds1# sh running-config interface gi1/0/20
interface gigabitethernet1/0/20
spanning-tree portfast
switchport access vlan 85

!
vcr1c1r2ds1# sh running-config interface gi2/0/20
interface gigabitethernet2/0/20
spanning-tree portfast
switchport access vlan 85

Here's the port config for the Dell switch ports connecting to the upstream Cisco Nexus switches:

interface tengigabitethernet1/0/1
channel-group 30 mode auto
!
interface tengigabitethernet1/0/2
channel-group 30 mode auto

interface tengigabitethernet2/0/1
channel-group 30 mode auto
!
interface tengigabitethernet2/0/2
channel-group 30 mode auto
!
interface Port-channel30
spanning-tree cost 10000
switchport mode trunk

Here's the port config for the Cisco Nexus switch ports connected to the downstream Dells (it's the same on both N9Ks):

interface Ethernet1/3
description VPC to Rack 2 5548s
switchport mode trunk
channel-group 102 mode active

interface Ethernet1/4
description VPC to Rack 2 5548s
switchport mode trunk
channel-group 102 mode active

interface port-channel102
description VPC to Rack 2 5548s
switchport mode trunk
vpc 102

Thanks

A