We use Cisco C9500-24Y4C stacked switches as our core routers to connect our different sites. We have a ring setup between our sites using dark fiber. We are using trunks with a VLAN as the L3 connection. The connection in question is also often the link that spanningtree most often blocks. We are using OSPF as our routing protocol. This setup has been running for at least 2 years flawlessly and was tested last month during a fiber cut due to road construction. We have 4 sites in this ring.
Yesterday, during maintenance, we found one of our layer 3 links down between 2 sites (2 & 4). Normally we are notified from Nagios but since the physical link is up, Nagios did not alert. Everything looks as it should. We even brought up another VLAN to do the L3 connection.
We noticed that if we shut the VLAN int on site 2, we can ping the IP on site 4. If the VLAN int on side 1 is no shut, we cannot ping the IP on side 2. We even tried to simulate a fiber cut by shutting another int in the ring and the L3 connection between site2 and site 4 did not come up.
sh ip int br shows the int up up on both sites
sh ip ospf nei does not show the connection between site 2 & 4
VLAN int and hu int from site 4
interface Vlan804
description 4_to_2
ip address 10.255.255.97 255.255.255.252
no ip redirects
ip pim sparse-dense-mode
ip ospf mtu-ignore
end
interface HundredGigE1/0/25
description Link to 2
switchport access vlan 804
switchport trunk allowed vlan 55,74-76,78-80,82,84,120,122,124,222,223,504,804
switchport trunk allowed vlan add 820
switchport mode trunk
mtu 9196
ip flow monitor nf-input input
ip flow monitor nf-output output
end
VLAN int and hu int from site 2
interface Vlan804
description 2_to_4
ip address 10.255.255.98 255.255.255.252
no ip redirects
ip pim sparse-dense-mode
ip ospf mtu-ignore
end
interface HundredGigE2/0/26
description Link to 4
switchport access vlan 804
switchport trunk allowed vlan 55,74-76,78-80,82,84,120,122,124,222,223,504,804
switchport trunk allowed vlan add 820
switchport mode trunk
mtu 9196
ip flow monitor nf-input input
ip flow monitor nf-output output
end
Does anyone have any insight into the issue?
