OSPF flapping while adding 9200 switches in Layer 2 network

[Deepak Kumar]

Hello Guys,

I am back here with a question. I am busy in my office work as well Cisco magic number study so time management is an issue for me.  Hoverwere I am facing a strange issue with Network as OSPF neighborship with SD-WAN devices flapping, once I will add new access switches Cisco 9200L. it is happing with 9200L switches only. I added 2960X switches but there is no issue.

we observed that this problem occurs with two configuration combinations such as 1. VLAN x (x = VLAN ID for OSPF neighborship only with SD-WAN device in P2P network) is the source of RSPAN (1) means making a duplicate copy of VLAN X traffic for security analysis by the security team. 2. VLAN X is allowed on the trunk ports of the newly added access switches (9200L switches). If we break any of a combination like remove VLAN X from RSPAN or Remove VLAN X from the trunk port of Cisco 9200L switches, the OSPF will stable again. I tried with different pair of new 9200L switches. Actually, we have 30 other switches and planning to add a few new 9200L switches. This issue is with all new Cisco. No routing is enabled on it. no client connected on any port on newly added switches.

Let me add a point here that dont go with best practice for allowing VLAN X on access switches trunk is good or not. It is not a design requirement but currently, design is like this only.

So far what is observed as an SD-WAN device is not getting hello message (issue with Multicast hello only) after adding new switches. Immediate Hello (unicast) is receiving by SD-WAN and trying to establish a neighborship again. The Core switch (4506x) is receiving a hello message without any issue. I can see that the Core switch is showing logs that he can't see himself in the neighborship list in the hello message hence dropping the neighborship and trying to established it again. 

We run multiple sessions with Cisco and internally. We didn't notice any issue with STP, or any other device trying to established a neighborship (unauthorized), DT has been disabled, 9200 switches are with fresh configuration (only AAA, and default added). 

The Firmware Version for 9200L is 16.12.4 and Core Switch is 3.8.x. 

The Cisco TAC is working on the case for the last 50 days but still no result. We also have a few limitations because we have the same setup on almost 90 locations with the same firmware version and this is the first location in which we started migration from static routing to OSPF. We can't change the firmware version until we will not get any technical proof (due to IT policy) and SD-WAN devices are not supported to non-broadcast network type to change hello to unicast.

Did you face the same type of issue? do you have any idea? 

[Deepak Kumar]

Hi,

Yes, you are right, Nothing is connected to C9200. RSPAN is configured on the core switch and the Destination Port is on the old C2960 switch.

C9200 running-configuration is attached.

[paul driver]

Hello Deepak

Thanks for sharing your config

As you dont have any edge ports open presently can you remove dhcp snooping ( which isnt required for all vlans anyway) plus IPDT which will turn off the ARP inspection that comes with it, reload the switches and test again.

no vtp interface Vlan11
no device-tracking policy DT_trunk_policy
no ip dhcp snooping vlan 1-4094
no ip dhcp snooping
spanning-tree mst 0 priority 61440

default interface TenGigabitEthernet1/1/1
interface TenGigabitEthernet1/1/1
description uplink_to_core
switchport mode trunk
spanning-tree link-type point-to-point
no shut

wr
reload

show spanning-tree
show spanning-tree mst
show spanning-tree mst interface ten1/1/1
sh vlan
sh int trunk

[Deepak Kumar]

Hi,

I didn't touch to DHCP snooping but IPDT is disabled in the last troubleshooting session only. There is no effect. But If I will disable the DHCP snooping then it will automatically disable the IPDT as well. I need to check with DHCP snooping. Do we have any technical justification to disable it in this case? I can't find it in my mind right now.

[paul driver]

Hello Deepak
I would say the technical justification to turn DHCP snooping off is it’s not being used on the 9200’s at present because you have no end users.

Also may I ask do you have snooping applied on the cores if so it should be turned off as it isn’t applicable at L3 and make sure you have manually specified stp p2p on the trunk towards the 9200 and from the 9200 towards the cores (shown previous example)

Lastly what version software do you have running on the 9200

[Deepak Kumar]

Hi,

DHCP snooping is not a core layer feature so it is not enabled on the core. 

Currently, I am in contact with the Cisco team and Cisco has been handover to Cisco TAC senior team (L3). I will monitor for two days and waiting for a reply. If there will no answer then I will try this weekend. 

[paul driver]

Hello Deepak
FYI -I'm well aware DHCP snooping isn’t a core feature but that doesn’t prohibit it from being applied, hence the query.
Another thing I’d like to mention is having dhcp snooping in path with your span sessions, that is also not suggested, But as those 9200 are doing basically nothing, I would suggest an ios upgrade if applicable.

Looking forward to what cisco tac comes back with as this has been a quite interesting discussion