Potential spanning tree issue

[andrewrocks]

Hi All,

I have a Cisco 3560 running as a Level3 device in my network running 10 VLANs and routing between most of them (nothing complex with ACLs) and running spanning-tree mode pvst. The main network is run on a netgear GS748TPS stack of three switches running MSTP.

I have just bought an additional 3560 and a 2960 to plug in. I have set them up with IP addresses and then plugged them into the netgear. This brought the whole network down until I unplugged the new switches.

I have confirmed the IP addresses aren't duplicated and that DHCP is not running on the switches so I can only assume it's something to do with DHCP. I cannot afford for the network to go offline again, so is there anything I should check? Am I running incompatible spanning tree methods between the netgear and cisco devices?

Thanks for any and all help.

Regards

Andrew.    

[Peter Paluch]

Hi Andrew,

To put it simply, all switches that support STP should run MSTP, configured identically with respect to the Configuration Name/Revision and mapping of VLANs into instances. Do not bother about switches that do not support STP at all.

Even though I am making this a little more complicated by requiring you to run MSTP everywhere if possible, I want to avoid some interoperability issues between Cisco's (R)PVST+ and pure IEEE MSTP run by your Netgear switches. At least we can be sure afterwards that the STP/MSTP is configured correctly.

Best regards,

Peter

[andrewrocks]

Hi Peter,

I've just been reading the release notes for firmware V5.0.0.23 (I am running v5.0.0.21) and came across this:

http://support.netgear.com/app/answers/detail/a_id/14737/related/1/session/L2F2LzEvdGltZS8xMzE5MDE1NjExL3NpZC9YdU9MbFhHaw%3D%3D

Fix the problem that GS7xxTPS with RSTP mode enabled does not handle  MSTP packets properly.  This may cause loop if connecting device is with  MSTP enabled.  

Does this sound like something that could be causing me issues here?

Regards

Andrew.

[Peter Paluch]

Hi Andrew,

The issue seems to be reversed: you are running MSTP and you potentially had issues with devices speaking STP and PVST+. However, technical descriptions of corrected bugs are not always exhaustive - I would certainly suggest upgrading your firmware. Perhaps this bug fix is relevant to us as well, although according to the description, it should not be.

Best regards,

Peter

[Julio Garcia]

Andrew,

I think you should stop running MST,  and go for simple RSTP.

reason being

1) you probably dont need to run MST  ,  how many vlans do you in network -- if its less than 20 ,  just use normal RSTP

2) Your network topology isnt particularly complex, i dont see any redundant links to cause loops

3) RSTP is easier to configure on both your Netgears and Ciscos.

4) Your problem solved quicker

[Peter Paluch]

Rob,

That was my original idea as well, and I was in fact waiting for somebody to point out that using plain RSTP would be easier

However, I am going for MSTP for two reasons:

1. Cisco switches are incapable of speaking pure RSTP on their trunk ports. They will always put their RPVST+ along with RSTP there. I am well aware of the principles of interoperation between RSTP and RPVST+ regions and still, I do not consider that to be a particularly bright idea. The MSTP is the only protocol that Cisco speaks in the same way as IEEE-compatible switches.
2. The network at Andrew's company may eventually grow larger and the count of VLANs may increase over time. Having the MSTP deployed now when the network is still simple will hopefully ease its potential future deployment.

Best regards,

Peter

[Alexander Maroukian]

Peter is absolutely right MSTP is in most of the cases pointed as solution in mixed vendor environment because of the reasons he mentioned.

Best regadrs,

Alex

[andrewrocks]

Hello All,

Right, after an early start I'm making some progress. I have setup the new cisco switches as you described and the netgear switches as you described. This is now working fine.

However, when I configured the current main (Layer 3) cisco with the settings you suggested I lost all communication with it and was forced to reboot it.

Do you have any suggestions as to why this occured? Also, now that everything is working, is there a danger with leaving the central switch running PVST?

Thanks again for your help

Andrew.

[Peter Paluch]

Andrew,

Is it possible for you to post the configuration of your main L3 switch? We are almost complete with deploying the MSTP and we should be able to get it completely working in your entire network.

Best regards,

Peter

[andrewrocks]

Hi Peter,

I agree that it is annoyingly close now and it would be great to get it to work.

My full config is:

ERSCoreSwitch#show running-config
Building configuration...
Current configuration : 10787 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ERSCoreSwitch
!
boot-start-marker
boot-end-marker
!
enable secret #################
enable password #############
!
!
!
no aaa new-model
clock summer-time UTC recurring last Sun Mar 1:00 last Sun Oct 2:00
system mtu routing 1500
ip routing
!
!
vtp mode transparent
!
!
crypto pki trustpoint TP-self-signed-2105128960
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-2105128960
 revocation-check none
 rsakeypair TP-self-signed-2105128960
!
!
crypto pki certificate chain TP-self-signed-2105128960
 certificate self-signed 01
 #############################
  quit
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
vlan 18
 name UKS_WAN
!
vlan 19
 name UKS_LAN
!
vlan 30
 name ERS_Public_WiFi
!
vlan 46
 name ERS_Legacy
!
vlan 101
 name Telewest_WAN
!
vlan 130
 name iSCSI
!
vlan 2000
 name Config
!
vlan 2001
 name ERS_Gateways
!
vlan 2010
 name ERS_Servers
!
vlan 2016
 name ERS_Network_Devices
!
vlan 2018
 name ERS_Telephone_Ctrl
!
vlan 2020
 name ERS_VServers
!
vlan 2051
 name ERS_TV_Studio_1
!
vlan 2052
 name ERS_TV_Studio_2
!
vlan 2053
 name ERS_TV_Studio_3
!
vlan 2055
 name ERS_TV_Matrix
!
vlan 2056
 name ERS_TV_CAR
!
vlan 2070
 name ERS_Desktops
!
vlan 2210
 name Unit43
!
!
!
interface FastEthernet0
 no ip address
 no ip route-cache cef
 no ip route-cache
 no ip mroute-cache
!
interface GigabitEthernet0/1
 description UL Stack sw LACP
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/2
 description UL Stack sw LACP
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/3
 description UL Stack sw LACP
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/4
 description iSCSI Uplink
 switchport access vlan 130
 switchport mode access
!
interface GigabitEthernet0/5
 description iSCSI Uplink
 switchport access vlan 130
 switchport mode access
!
interface GigabitEthernet0/6
 description UL Warehouse
 switchport access vlan 46
 switchport mode access
!
interface GigabitEthernet0/7
 description UL TV CAR
 switchport access vlan 2056
 switchport mode access
!
interface GigabitEthernet0/8
 description UL TV Studio 1
 switchport access vlan 2051
 switchport mode access
!
interface GigabitEthernet0/9
 description UL TV Studio 2
 switchport access vlan 2052
 switchport mode access
!
interface GigabitEthernet0/10
 description UL TV Studio 3
 switchport access vlan 2053
 switchport mode access
!
interface GigabitEthernet0/11
 description UL Unit 43
 switchport access vlan 46
 switchport mode access
!
interface GigabitEthernet0/12
 description ERS_Legacy_46
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/13
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
 speed 1000
 duplex full
!
interface GigabitEthernet0/14
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/15
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/16
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/17
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/18
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/19
 description TelephoneServers
 switchport access vlan 2018
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode access
!
interface GigabitEthernet0/20
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/21
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/22
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/23
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/24
 description TelephoneServers
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/25
 description Draytek 1
 switchport access vlan 2001
 switchport mode access
!
interface GigabitEthernet0/26
 description Draytek 2
 switchport access vlan 2001
 switchport mode access
!
interface GigabitEthernet0/27
 description Censornet
 switchport access vlan 2001
 switchport mode access
!
interface GigabitEthernet0/28
 switchport access vlan 46
 switchport mode access
!
interface GigabitEthernet0/29
 description Xenserver 101 Nic0
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/30
 description Xenserver 101 Nic1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/31
 description Xenserver 200 Nic0
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/32
 description Xenserver 200 Nic1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/33
 description Xenserver 200 Nic2
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/34
 description Xenserver 200 Nic3
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport mode trunk
!
interface GigabitEthernet0/35
 description Eagle 6
 switchport access vlan 2010
 switchport trunk native vlan 2020
 switchport mode access
!
interface GigabitEthernet0/36
 description Xenserver 201 Nic1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 2020
 switchport mode trunk
!
interface GigabitEthernet0/37
 description Xenserver 201 Nic2
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 2020
 switchport mode trunk
!
interface GigabitEthernet0/38
 description Xenserver 201 Nic3
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 2020
 switchport mode trunk
!
interface GigabitEthernet0/39
!
interface GigabitEthernet0/40
!
interface GigabitEthernet0/41
 description ConfigPC
 switchport access vlan 2000
 switchport mode access
!
interface GigabitEthernet0/42
 description Uplink to Telephone Network
 switchport access vlan 2018
 switchport mode access
!
interface GigabitEthernet0/43
 description Legacy Phone System
 switchport access vlan 46
!
interface GigabitEthernet0/44
 description Uplink to UKS
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 46
 switchport trunk allowed vlan 18,19,46,130
 switchport mode trunk
!
interface GigabitEthernet0/45
 description Draytek1_WAN1
 switchport access vlan 101
 switchport mode access
!
interface GigabitEthernet0/46
 description Draytek2_WAN1
 switchport access vlan 101
 switchport mode access
!
interface GigabitEthernet0/47
 switchport access vlan 101
 switchport mode access
!
interface GigabitEthernet0/48
 description WAN_Telewest
 switchport access vlan 101
 switchport mode access
!
interface GigabitEthernet1/1
!
interface GigabitEthernet1/2
!
interface GigabitEthernet1/3
!
interface GigabitEthernet1/4
!
interface TenGigabitEthernet1/1
!
interface TenGigabitEthernet1/2
!
interface Vlan1
 no ip address
!
interface Vlan18
 no ip address
!
interface Vlan19
 ip address 172.19.0.1 255.255.0.0
 ip helper-address 172.20.10.255
!
interface Vlan46
 ip address 192.168.46.1 255.255.254.0
 ip helper-address 172.20.10.255
!
interface Vlan101
 description Telewest_WAN
 no ip address
!
interface Vlan2000
 ip address 172.20.0.1 255.255.255.0
!
interface Vlan2001
 ip address 172.20.1.1 255.255.255.0
!
interface Vlan2010
 ip address 172.20.10.1 255.255.255.0
!
interface Vlan2012
 no ip address
!
interface Vlan2014
 ip address 172.20.14.1 255.255.255.0
 ip helper-address 172.20.10.255
!
interface Vlan2016
 no ip address
 ip helper-address 172.20.10.255
!
interface Vlan2018
 ip address 172.20.18.1 255.255.255.0
!
interface Vlan2020
 ip address 172.20.20.1 255.255.255.0
 ip helper-address 172.20.10.255
!
interface Vlan2051
 ip address 172.20.51.1 255.255.255.0
 ip helper-address 172.20.10.255
!
interface Vlan2052
 ip address 172.20.52.1 255.255.255.0
 ip helper-address 172.20.10.255
!
interface Vlan2053
 ip address 172.20.53.1 255.255.255.0
 ip helper-address 172.20.10.255
!
interface Vlan2055
 ip address 172.20.55.1 255.255.255.0
!
interface Vlan2056
 ip address 172.20.56.1 255.255.254.0
!
interface Vlan2070
 ip address 172.20.70.1 255.255.254.0
 ip helper-address 172.20.10.255
!
interface Vlan2210
 ip address 172.22.10.1 255.255.255.0
 ip helper-address 172.20.10.255
!
interface Vlan2220
 no ip address
!
ip classless
no ip forward-protocol udp
ip route 0.0.0.0 0.0.0.0 172.20.1.10
ip http server
ip http secure-server
!
ip sla enable reaction-alerts
snmp-server community public RO
snmp-server location ERS CAR
snmp-server contact IT
!
!
line con 0
line vty 0 4
 password ############
 login
line vty 5 15
 password ############
 login
!
end

Cheers

Andrew.

[Peter Paluch]

Hello Andrew,

Alright, thank you for the config.

I suggest proceeding as follows: First, add the following configuration snippet to your central 3560 config:

configure terminal
 spanning-tree mst 0 priority 4096
 spanning-tree mst configuration
  name MST
  revision 1
  instance 0 vlan 1-4094
 exit
end 

This configuration should make your MSTP configuration to be identical with surrounding switches, assuming they are configured identically (this must be ensured by you - Configuration Name = MST, Configuration Revision = 1, only a single instance 0 containing all VLANs; if there are other MST instances, they are supposed to be empty, i.e. holding no VLANs).

Adding these lines will not cause any interruption to your network because they only add parameters for MSTP which is not currently running on your 3560. These modifications are safe, then.

Next, in a period of low network activity, add these commands to your 3560:

configure terminal
 spanning-tree mode mst
end

This will activate the MSTP with the settings you have placed into the configuration beforehand. If this modification causes a network outage please record the output of all following commands (the entire output - use the capture function in your PuTTY/HyperTerminal software):

terminal length 0

show spanning-tree

show spanning-tree mst

show spanning-tree mst config

show spanning-tree mst config digest

show spanning-tree root

show spanning-tree blockedports

show spanning-tree inconsistentports

Then enter the following commands to restore the PVST back (of course, do not enter them if the MSTP appears to work correctly):

configure terminal
 spanning-tree mode pvst
end

Thank you!

Best regards

Peter

[andrewrocks]

Hi Peter,

This is probably a dumb question, but how do I change it back to pvst if I've been kicked out due to spanning tree?

Regards

Andrew.

[Alexander Maroukian]

Hi,

Peter has wirtten it in the end:

"Then enter the following commands to restore the PVST back (of course, do not enter them if the MSTP appears to work correctly):

configure terminal
 spanning-tree mode pvst
end
"

Best regards,
Alex 

[Peter Paluch]

Alex,

Thank you very much!

Andrew, one more thing: I probably forgot to stress that it would be best to enter the commands I have suggested directly via the console on your central 3560. Do not use Telnet/SSH if possible to configure these things. The reason is that if you experience another outage for whatever reason, you may not be able to access the 3560 over the network anymore. Using the console in this case is a safe bet.

Best regards,

Peter

[andrewrocks]

Hi Peter,

This answers my question. I did mean "seeing as last time I lost access to the switch, how do I execute the command to put it back" but the console is the obvious answer (d'oh!).

I will be trying this in the next few days (have to wait for right time in the week).

Regards

Andrew.