Re: Layer 3 connection stopped logically connecting

mccoyb_scc · ‎06-22-2023

We use Cisco C9500-24Y4C stacked switches as our core routers to connect our different sites. We have a ring setup between our sites using dark fiber. We are using trunks with a VLAN as the L3 connection. The connection in question is also often the link that spanningtree most often blocks. We are using OSPF as our routing protocol. This setup has been running for at least 2 years flawlessly and was tested last month during a fiber cut due to road construction. We have 4 sites in this ring.

Yesterday, during maintenance, we found one of our layer 3 links down between 2 sites (2 & 4). Normally we are notified from Nagios but since the physical link is up, Nagios did not alert. Everything looks as it should. We even brought up another VLAN to do the L3 connection.

We noticed that if we shut the VLAN int on site 2, we can ping the IP on site 4. If the VLAN int on side 1 is no shut, we cannot ping the IP on side 2. We even tried to simulate a fiber cut by shutting another int in the ring and the L3 connection between site2 and site 4 did not come up.

sh ip int br shows the int up up on both sites

sh ip ospf nei does not show the connection between site 2 & 4

VLAN int and hu int from site 4

interface Vlan804
description 4_to_2
ip address 10.255.255.97 255.255.255.252
no ip redirects
ip pim sparse-dense-mode
ip ospf mtu-ignore
end

interface HundredGigE1/0/25
description Link to 2
switchport access vlan 804
switchport trunk allowed vlan 55,74-76,78-80,82,84,120,122,124,222,223,504,804
switchport trunk allowed vlan add 820
switchport mode trunk
mtu 9196
ip flow monitor nf-input input
ip flow monitor nf-output output
end

VLAN int and hu int from site 2

interface Vlan804
description 2_to_4
ip address 10.255.255.98 255.255.255.252
no ip redirects
ip pim sparse-dense-mode
ip ospf mtu-ignore
end

interface HundredGigE2/0/26
description Link to 4
switchport access vlan 804
switchport trunk allowed vlan 55,74-76,78-80,82,84,120,122,124,222,223,504,804
switchport trunk allowed vlan add 820
switchport mode trunk
mtu 9196
ip flow monitor nf-input input
ip flow monitor nf-output output
end

Does anyone have any insight into the issue?

Reza Sharifi · ‎06-22-2023

Hi,

Can you post the OSPF configuration from both sides? Also, if you are planning on using VLAN 804 as a transit vlan between site-2 and 4, there is no need for trunk ports with all the VLANs in it, all you need is an access port with IP. See the example below:

interface Vlan804
description 2_to_4
ip address 10.255.255.98 255.255.255.252
no ip redirects
ip pim sparse-dense-mode
ip ospf mtu-ignore

interface HundredGigE2/0/26
description Link to 4
switchport access vlan 804
switchport mode access
mtu 9196
ip flow monitor nf-input input
ip flow monitor nf-output output

mccoyb_scc · ‎06-22-2023

Thank Reza. We need the trunk to forward the VLANs. I know this is not preferred but it is a solution that was needed.

Site 2

router ospf 101
router-id 2.1.1.1
auto-cost reference-bandwidth 40000
network 10.2.32.0 0.0.0.255 area 2
network 10.255.255.28 0.0.0.3 area 0
network 10.255.255.96 0.0.0.3 area 0
network 172.20.8.0 0.0.0.63 area 2
network 172.20.8.64 0.0.0.63 area 2
network 192.168.20.0 0.0.0.255 area 2
network 192.168.21.0 0.0.0.255 area 2
network 192.168.22.0 0.0.0.255 area 2
network 192.168.23.0 0.0.0.255 area 2
network 192.168.24.0 0.0.0.255 area 2
network 192.168.25.0 0.0.0.255 area 2
network 192.168.26.0 0.0.0.255 area 2
network 192.168.27.0 0.0.0.255 area 2
network 192.168.28.0 0.0.0.255 area 2
network 192.168.29.0 0.0.0.255 area 2
network 192.168.30.0 0.0.0.255 area 2
network 192.168.32.0 0.0.0.255 area 2
network 192.168.33.0 0.0.0.255 area 2
network 192.168.34.0 0.0.0.255 area 2
network 192.168.35.0 0.0.0.255 area 2
network 192.168.37.0 0.0.0.255 area 2
network 192.168.38.0 0.0.0.255 area 2
network 192.168.39.0 0.0.0.255 area 2
network 192.168.40.0 0.0.0.255 area 2
network 192.168.41.0 0.0.0.255 area 2
network 192.168.43.0 0.0.0.255 area 2
network 192.168.0.0 0.0.255.255 area 0

Site 4

router ospf 101
router-id 4.1.1.1
auto-cost reference-bandwidth 40000
network 10.4.0.0 0.0.255.255 area 4
network 10.254.67.0 0.0.0.63 area 4
network 10.255.255.36 0.0.0.3 area 0
network 10.255.255.96 0.0.0.3 area 0
network 10.255.255.104 0.0.0.3 area 0
network 172.20.9.0 0.0.0.63 area 4
network 172.20.9.64 0.0.0.63 area 4
network 192.168.160.0 0.0.0.255 area 4
network 192.168.161.0 0.0.0.255 area 4
network 192.168.162.0 0.0.0.255 area 4
network 192.168.163.0 0.0.0.255 area 4
network 192.168.164.0 0.0.0.255 area 4
network 192.168.165.0 0.0.0.255 area 4
network 192.168.166.0 0.0.0.255 area 4
network 192.168.167.0 0.0.0.255 area 4
network 192.168.168.0 0.0.0.255 area 4
network 192.168.169.0 0.0.0.255 area 4
network 192.168.170.0 0.0.0.255 area 4
network 192.168.171.0 0.0.0.255 area 4
network 192.168.172.0 0.0.0.255 area 4
network 192.168.173.0 0.0.0.255 area 4
network 192.168.174.0 0.0.0.255 area 4
network 192.168.175.0 0.0.0.255 area 4
network 192.168.176.0 0.0.0.255 area 4
network 192.168.177.0 0.0.0.255 area 4
network 192.168.179.0 0.0.0.255 area 4
network 192.168.180.0 0.0.0.255 area 4
network 192.168.0.0 0.0.255.255 area 0

Reza Sharifi · ‎06-22-2023

network 10.255.255.96 0.0.0.3 area 0

Do you see an OSPF neighborship between the 2 routers using the above subnet? or any neighborship at all?

Can you put a quick drawing together showing how everything is connected and post it here?

HTH

mccoyb_scc · ‎06-22-2023

All other L3 links are up and working configured the same (except VLAN and IP).

Reza Sharifi · ‎06-22-2023

Just to make sure there is no MTU mismatch, can you delete "ip ospf mtu-ignore" and "mtu 9196" on both routers and check the neighborship again?

HTH

mccoyb_scc · ‎06-22-2023

I did so. the OSPF neighbor did not come up

OV-Core-9500#sh ip os nei

Neighbor ID Pri State Dead Time Address Interface
1.1.1.1 1 FULL/BDR 00:00:36 10.255.255.29 Vlan501

ALT-Core-9500(config-if)#do sh ip os nei

Neighbor ID Pri State Dead Time Address Interface
3.1.1.1 1 FULL/BDR 00:00:35 10.255.255.37 Vlan503

mccoyb_scc · ‎06-22-2023

I guess I should add that router 2.1.1.1 or 4.1.1.1 should be a neighbor.

MHM Cisco World · ‎06-22-2023

I will check your topology

Reza Sharifi · ‎06-22-2023

I guess I should add that router 2.1.1.1 or 4.1.1.1 should be a neighbor.

Correct. If you are doing this between 2 and 4 then the router ids should not be 1.1.1.1 1 and 3.1.1.1.1

Reza Sharifi · ‎06-22-2023

Are all the 9500 configured as StackWise Virtual?

mccoyb_scc · ‎06-22-2023

Yes.

ALT-Core-9500#sh stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 1

Switch Stackwise Virtual Link Ports
------ ---------------------- ------
1 1 HundredGigE1/0/27
HundredGigE1/0/28
2 1 HundredGigE2/0/27
HundredGigE2/0/28

OV-Core-9500#sh stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 1

Switch Stackwise Virtual Link Ports
------ ---------------------- ------
1 1 HundredGigE1/0/27
HundredGigE1/0/28
2 1 HundredGigE2/0/27
HundredGigE2/0/28

MHM Cisco World · ‎06-22-2023

there is L2 Loop and SPT must BLK one link, the STP BLK link between site2 and site4 ?

Now share show ip ospf interface in all sites

mccoyb_scc · ‎06-22-2023

STP is blocking this link. Hu1/0/25 Altn BLK 500 128.217 P2p

The files are attached.

MHM Cisco World · ‎06-22-2023

show ip ospf interface brief <<-