OSPF behaviour during core change

[stuart_jones]

Hi all,

During a recent core swap out we experienced some routed traffic being black holed. This happened when the layer 2 connections were brought up on the new core and was resolved when the layer 3 interfaces were brought up to form the OSPF neighbourships.

The general setup is two core L3 switches with L3 distribution pairs hanging off them.  Each device has a P2P OSPF SVI that reaches its neighbouring device over a L2 port-channel.

When swapping out core 1 the old device was completely powered down. The new device was then powered up and the port-channels were brought up before the L3 interfaces, this would have brought up the SVI on the distribution switch (as there was now an interface configured with the VLAN) for the corresponding SVI on core 1 which was still down.

At this stage any particular distribution switch would have its P2P OSPF SVI up and neighbourship up with core 2. The P2P OSPF SVI for core 1 would be up but the neighbourship with core 1 would not be as the SVI was down on core 1.

Each distribution switch also has two default routes configured, one for core 1 and one for core 2.

The traffic being black holed had its HSRP gateway on the distribution switch and was destined for the internet.

Any ideas as to why this might have happened? If any of the above isn’t clear please let me know.

Thanks in advance,

Stuart

[Georg Pauwen]

Hello,

it is a bit hard to visualize your topology, can you post a schematic drawing with all devices involved, and indicate which device was swapped ?

[Jon Marshall]

So each distribution switch has 2 default routes pointing to each core switch and these are not OSPF routes by the sounds of it. 

What are the next hop IPs for the static routes ? 

Jon

[stuart_jones]

Hi Jon,

Thanks for your reply.

The two static routes point to the P2P OSPF SVIs on the core 1 and core 2 of which the core 1 SVI was down. As the distribution SVI came up with the port channel the static route would have become a valid route again?

[Giuseppe Larosa · Accepted solution]

Hello @stuart_jones ,

>> Each distribution switch also has two default routes configured, one for core 1 and one for core 2.

Do you mean that you had static default routes configured on each distribution device as noted by @Jon Marshall ?

If so, when the P2P SVI with new core1 comes up if the distribution has still an ARP entry for the core1 SVI IP address used as next-hop in the static default route you have created a black hole because that static route is considered valid and used.

You should have the core routers to generate a default route in OSPF using the

router ospf 10

default-information originate

instead of the static routes on each distribution device.

Hope to help

Giuseppe

[stuart_jones]

Hi Giuseppe,

Thanks for your reply.

As I replied to Jon the static routes go back to both cores. You're right that we shouldn't use static default routes on the distributions. This is our legacy network which we're looking to move away from so that will be resolved in the future.

Regards,
Stuart