02-03-2014 01:32 AM - edited 03-07-2019 05:57 PM
Hi,
I'm working for a company that has 2x 6500 chasis switches in the main building as Core switches (CORE1 and CORE2). There are 3 other buildings that house employees (Building 2 and Building 3) and a DR site. The "Core" switches at these other buildings are 3750 switches (stacks of 2). The buildings are connected with 1Gb fibre (MM) leased lines in a square:
Since a few days we are seeing alot of spanning tree recalculations on the Core switches of Building 2 and 3 which causes alot of network issues for the people in those buildings. More precisely the Gi1/0/1 interface on both core switches of those buildings (see red crosses in picture) are constantly displaying these messages:
Feb 3 10:25:31 Building2-CORE 801113: 690303: Feb 3 10:24:20.544 cet: RSTP(750): Gi1/0/1 rcvd info expired
Feb 3 10:25:31 Building2-CORE 801114: 690304: Feb 3 10:24:20.544 cet: %SPANTREE-2-LOOPGUARD_BLOCK: Loop guard blocking port GigabitEthernet1/0/1 on VLAN0750.
Feb 3 10:25:32 Building2-CORE 801115: 690305: Feb 3 10:24:20.544 cet: RSTP(750): updt roles, information on root port Gi1/0/1 expired
Feb 3 10:25:32 Building2-CORE 801116: 690306: Feb 3 10:24:20.544 cet: RSTP(750): we become the root bridge
Feb 3 10:25:32 Building2-CORE 801117: 690307: Feb 3 10:24:20.552 cet: RSTP(750): updt roles, received superior bpdu on St1
Feb 3 10:25:32 Building2-CORE 801118: 690308: Feb 3 10:24:20.552 cet: RSTP(750): St1 is now root port
Feb 3 10:25:32 Building2-CORE 801119: 690309: Feb 3 10:24:20.552 cet: RSTP(750): synced St1
Feb 3 10:25:32 Building2-CORE 801120: 690310: Feb 3 10:24:20.561 cet: RSTP(750): transmitting an agreement on St1 as a response to a proposal
Feb 3 10:26:21 Building2-CORE 801193: 690383: Feb 3 10:25:10.910 cet: %SPANTREE-2-LOOPGUARD_UNBLOCK: Loop guard unblocking port GigabitEthernet1/0/1 on VLAN0750.
Feb 3 10:26:21 Building2-CORE 801194: 690384: Feb 3 10:25:10.910 cet: RSTP(750): initializing port Gi1/0/1
Feb 3 10:26:21 Building2-CORE 801195: 690385: Feb 3 10:25:10.910 cet: RSTP(750): Gi1/0/1 is now designated
Feb 3 10:26:21 Building2-CORE 801196: 690386: Feb 3 10:25:10.910 cet: RSTP(750): updt roles, received superior bpdu on Gi1/0/1
Feb 3 10:26:21 Building2-CORE 801197: 690387: Feb 3 10:25:10.910 cet: RSTP(750): Gi1/0/1 is now root port
Feb 3 10:26:21 Building2-CORE 801198: 690388: Feb 3 10:25:10.910 cet: RSTP(750): St1 blocked by re-root
Feb 3 10:26:21 Building2-CORE 801199: 690389: Feb 3 10:25:10.910 cet: RSTP(750): St1 is now designated
Feb 3 10:26:21 Building2-CORE 801209: 690399: Feb 3 10:25:10.919 cet: RSTP(750): transmitting a proposal on St1
Feb 3 10:26:21 Building2-CORE 801211: 690401: Feb 3 10:25:10.927 cet: RSTP(750): synced Gi1/0/1
Feb 3 10:26:22 Building2-CORE 801212: 690402: Feb 3 10:25:10.927 cet: RSTP(750): received an agreement on St1
And less than a minute later the same again. This is happening with all VLANs. There's about 125 VLANs and all go over the square.
From what I understand this means BPDU packts are not received in time (2 seconds) and spanning tree starts recalculation. We already asked the provider of the leased lines to test them but they claim nothing is wrong with them. It"s also a bit weird that we are seeing this on 2 different places (physically different locations and lines).
CPU usage looks normal (around 14%) on all switches in this square. Since it's happening on 2 locations I don't think a faulty cable or SFP is causing this.
Any ideas from you guys?
Regards
02-03-2014 01:57 AM
Hi
Can you show us the config on the 6500 core switch?
Presumably the links between the buildings are Trunks?
Can post results of the following command on each set of switches:
#show spanning tree vlan 750
02-03-2014 02:26 AM
Hi,
All links between the buildings are configured as trunks indeed with no VLAN restrictions (all VLANs allowed).
Here is the extract of the command on all 5 switches/stacks:
MAIN-CORE1#sh spanning-tree vlan 750
VLAN0750
Spanning tree enabled protocol rstp
Root ID Priority 8192
Address 001c.0edc.eaee
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 8192
Address 001c.0edc.eaee
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300
Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/3 Desg FWD 4 128.3 P2p
Gi1/4 Desg FWD 4 128.4 P2p
Gi1/5 Desg FWD 4 128.5 P2p
Gi1/6 Desg FWD 4 128.6 P2p
Gi1/7 Desg FWD 4 128.7 P2p
Gi2/22 Desg FWD 4 128.150 P2p
Gi2/23 Desg FWD 4 128.151 P2p
Po10 Desg FWD 3 128.1666 P2p
Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po11 Desg FWD 3 128.1667 P2p
MAIN-CORE2#sh spanning-tree vlan 750
VLAN0750
Spanning tree enabled protocol rstp
Root ID Priority 8192
Address 001c.0edc.eaee
Cost 3
Port 1666 (Port-channel10)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 16384
Address 001c.0edc.daee
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300
Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/3 Desg FWD 4 128.3 P2p
Gi1/4 Desg FWD 4 128.4 P2p
Gi1/5 Desg FWD 4 128.5 P2p
Gi1/6 Desg FWD 4 128.6 P2p
Gi1/9 Desg FWD 4 128.9 P2p
Po10 Root FWD 3 128.1666 P2p
Po21 Desg FWD 4 128.1667 P2p
Building2-CORE1#show spanning-tree vlan 750
VLAN0750
Spanning tree enabled protocol rstp
Root ID Priority 8192
Address 001c.0edc.eaee
Cost 7
Port 1 (GigabitEthernet1/0/1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 33518 (priority 32768 sys-id-ext 750)
Address 108c.cf03.1d00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi1/0/1 Root FWD 4 128.1 P2p
St1 Desg FWD 100 128.872 P2p
Gi2/0/1 Desg FWD 4 128.55 P2p
Building3-CORE1#show spanning-tree vlan 750
VLAN0750
Spanning tree enabled protocol rstp
Root ID Priority 8192
Address 001c.0edc.eaee
Cost 11
Port 55 (GigabitEthernet2/0/1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 33518 (priority 32768 sys-id-ext 750)
Address 8cb6.4fb9.7300
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi1/0/1 Root BKN*4 128.1 P2p *LOOP_Inc
St1 Root FWD 100 128.872 P2p
Gi2/0/1 Root FWD 4 128.55 P2p
DR-01#show spanning-tree vlan 750
VLAN0750
Spanning tree enabled protocol rstp
Root ID Priority 8192
Address 001c.0edc.eaee
Cost 4
Port 54 (GigabitEthernet2/0/2)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 33518 (priority 32768 sys-id-ext 750)
Address 0013.c37a.e300
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi2/0/2 Root FWD 4 128.54 P2p
Gi1/0/1 Desg FWD 4 128.1 P2p
Fa1/0/13 Desg FWD 19 128.15 P2p
Here is the config of MAIN-CORE1 (I removed most interfaces, VLAN interfaces and ACL's from it):
MAIN-CORE1#sh run
Building configuration...
Current configuration : 44402 bytes
!
upgrade fpd auto
version 12.2
no service pad
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
service counters max age 5
!
hostname MAIN-CORE1
!
boot-start-marker
boot system flash sup-bootdisk:s72033-ipservicesk9-vz.122-33.SXI6.bin
boot system flash sup-bootdisk:s72033-ipservicesk9-vz.122-18.SXF8.bin
boot-end-marker
!
security passwords min-length 1
logging buffered 5000000
no logging console
no logging monitor
!
aaa new-model
!
!
aaa authentication login default group radius local
aaa authentication login CONSOLE local
aaa authentication dot1x default group radius
aaa authorization exec default group radius local
aaa authorization network default group radius local
!
!
!
aaa session-id common
clock timezone cet 1
clock summer-time CEST recurring last Sun Mar 2:00 last Sun Oct 3:00
!
!
no ip domain-lookup
ip tftp source-interface Vlan60
!
ip ftp source-interface Vlan60
ip flow ingress layer2-switched vlan 20
ip sla 3
icmp-echo 172.31.99.5 source-ip X.X.X.X
timeout 2000
frequency 5
ip sla schedule 3 life forever start-time now
ip sla 4
icmp-echo X.X.X.X source-ip X.X.X.X
frequency 5
ip sla schedule 4 life forever start-time now
udld aggressive
udld message time 7
mls qos map cos-dscp 0 10 18 24 34 46 48 56
mls qos
mls netflow interface
no mls acl tcam share-global
mls cef error action freeze
!
!
!
!
!
!
!
errdisable recovery cause udld
errdisable recovery cause security-violation
errdisable recovery cause psecure-violation
errdisable recovery interval 30
diagnostic bootup level minimal
!
spanning-tree mode rapid-pvst
spanning-tree vlan 1,21,166,168,210,842-843 priority 16384
spanning-tree vlan 2-3,7,10,17-18,28,41,44,60,70,78,96,110,112 priority 8192
spanning-tree vlan 121-122,125,127,140,169-170,199,209,213-214 priority 8192
spanning-tree vlan 220-221,253-254,299,318-322,343,350,411,415 priority 8192
spanning-tree vlan 420-421,425,430,450-451,460,500-501,540,602 priority 8192
spanning-tree vlan 650,702,710-716,740,750,895,900-902,910,920 priority 8192
spanning-tree vlan 940 priority 8192
spanning-tree vlan 20 priority 9
spanning-tree vlan 40 priority 8191
!
redundancy
main-cpu
auto-sync running-config
mode sso
!
vlan internal allocation policy ascending
vlan access-log ratelimit 2000
!
class-map match-any test
class-map match-all DoubleTake_map
match access-group name DoubleTake
!
!
policy-map DoubleTake_Pol
class DoubleTake_map
set ip dscp af41
!
interface Port-channel10
description connection between cores
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
mls qos trust cos
!
interface GigabitEthernet1/3
description Trunk To access-sw1
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 17,20,100,112,140,209,300,740,750
switchport mode trunk
switchport nonegotiate
mls qos trust cos
!
interface GigabitEthernet1/4
description Trunk To access-sw2
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 17,20,27,100,112,209,740,750
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet1/5
description Trunk To access-sw3
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 17,20,70,112,209,221,740,750,901,902
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet1/6
description Trunk To access-sw4
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,17,20,28,60,70,100,112,140,209,220,300,343
switchport trunk allowed vlan add 350,540,602,640,641,740,750,840-842,902
switchport mode trunk
switchport nonegotiate
mls qos trust cos
!
interface GigabitEthernet1/7
description Trunk to DR
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
speed nonegotiate
mls qos trust cos
!
interface GigabitEthernet2/22
description Link to FW1
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,40,165,211-214,220,318,420,451,501,650,651
switchport trunk allowed vlan add 750
switchport mode trunk
logging event link-status
logging event spanning-tree status
load-interval 30
!
interface GigabitEthernet2/23
description link to FW1
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 78,121,122,124-127,221,319-322,411,415,425,430
switchport trunk allowed vlan add 450,460,461,465,602,712,713,716,750
switchport mode trunk
logging event link-status
logging event spanning-tree status
load-interval 30
mls qos trust dscp
spanning-tree portfast edge
!
interface GigabitEthernet5/1
description Trunk To MAIN-CORE2
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
mls qos trust cos
channel-group 10 mode on
!
interface GigabitEthernet5/2
description Trunk To MAIN-CORE2
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
mls qos trust cos
channel-group 10 mode on
!
ip default-gateway X.X.X.X
ip classless
ip forward-protocol nd
ip forward-protocol udp discard
ip route X.X.X.X Y.Y.Y.Y
!
ip http server
ip http access-class 39
ip http authentication local
no ip http secure-server
ip flow-export source Vlan20
ip flow-export version 9
ip flow-export destination X.X.X.X 2000
!
!
ip radius source-interface Vlan20
logging trap debugging
logging source-interface Vlan20
logging X.X.X.X
!
tftp-server sup-bootdisk:s72033-ipservicesk9-vz.122-33.SXH1.bin
snmp-server community X
snmp-server ifindex persist
snmp ifmib ifindex persist
!
radius-server host X.X.X.X. auth-port 1645 acct-port 1646 key 7 Y
radius-server host X.X.X.X auth-port 1645 acct-port 1646 key 7 Y
!
control-plane
!
!
dial-peer cor custom
!
line con 0
exec-timeout 20 0
privilege level 15
password 7 Y
logging synchronous
login authentication CONSOLE
stopbits 1
line vty 0 4
session-timeout 300
access-class vty_mgmt in
transport input telnet
line vty 5 15
session-timeout 60
access-class vty_mgmt in
transport input telnet
!
exception core-file
mac-address-table notification mac-move
ntp clock-period 17179825
ntp source Vlan20
ntp master 1
!
end
02-03-2014 02:50 AM
What is st1 connected to in buidings 2 & 3 ?
Jon
02-03-2014 04:15 AM
Hi,
I believe it's the stack port. Buildings 2 and 3 boh have 2x WS-C3750G-12S switches in a stack.
The DR site is also a stack (2x WS-C3750-24TS) but does not show the St1 port in the spanning tree info. I'm not sure why.
Regards
02-04-2014 08:24 AM
Hello, Esger.
I guess it could be a scaling issue on your 6500.
Please refer to http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_Infra2_5/DCInfra_5.html
Pay attention to "Virtual port per Line card" limitations - it's a valid concern for your design, as you are not pruning VLANs on trunks.
In the configuration provided, you are using about 70 VLANs, so limitation of 1800 virtual ports per line card will be hit with 1800/70=25 ports configured as trunk on the same line card.
PS: to check stability of your fiber, try to use udld with tuned timers (less than STP max_age timer).
PS2: I would build L3 links between buildings, unless you desperately need VLAN extension between buildings.
PS2 added
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide