Re: Layer 3 connection stopped logically connecting - Page 2

mccoyb_scc · ‎06-22-2023

We use Cisco C9500-24Y4C stacked switches as our core routers to connect our different sites. We have a ring setup between our sites using dark fiber. We are using trunks with a VLAN as the L3 connection. The connection in question is also often the link that spanningtree most often blocks. We are using OSPF as our routing protocol. This setup has been running for at least 2 years flawlessly and was tested last month during a fiber cut due to road construction. We have 4 sites in this ring.

Yesterday, during maintenance, we found one of our layer 3 links down between 2 sites (2 & 4). Normally we are notified from Nagios but since the physical link is up, Nagios did not alert. Everything looks as it should. We even brought up another VLAN to do the L3 connection.

We noticed that if we shut the VLAN int on site 2, we can ping the IP on site 4. If the VLAN int on side 1 is no shut, we cannot ping the IP on side 2. We even tried to simulate a fiber cut by shutting another int in the ring and the L3 connection between site2 and site 4 did not come up.

sh ip int br shows the int up up on both sites

sh ip ospf nei does not show the connection between site 2 & 4

VLAN int and hu int from site 4

interface Vlan804
description 4_to_2
ip address 10.255.255.97 255.255.255.252
no ip redirects
ip pim sparse-dense-mode
ip ospf mtu-ignore
end

interface HundredGigE1/0/25
description Link to 2
switchport access vlan 804
switchport trunk allowed vlan 55,74-76,78-80,82,84,120,122,124,222,223,504,804
switchport trunk allowed vlan add 820
switchport mode trunk
mtu 9196
ip flow monitor nf-input input
ip flow monitor nf-output output
end

VLAN int and hu int from site 2

interface Vlan804
description 2_to_4
ip address 10.255.255.98 255.255.255.252
no ip redirects
ip pim sparse-dense-mode
ip ospf mtu-ignore
end

interface HundredGigE2/0/26
description Link to 4
switchport access vlan 804
switchport trunk allowed vlan 55,74-76,78-80,82,84,120,122,124,222,223,504,804
switchport trunk allowed vlan add 820
switchport mode trunk
mtu 9196
ip flow monitor nf-input input
ip flow monitor nf-output output
end

Does anyone have any insight into the issue?

mccoyb_scc · ‎06-22-2023

Site 1

Interface PID Area IP Address/Mask Cost State Nbrs F/C
Twe1/0/13 101 0 10.255.255.45/30 4 DR 1/1
Vl515 101 0 10.255.255.77/30 40 DR 1/1
Vl254 101 0 192.168.254.1/24 40 DOWN 0/0
Vl212 101 0 192.168.212.1/24 40 DR 0/0
Vl206 101 0 192.168.206.1/24 40 DR 0/0
Vl205 101 0 192.168.205.1/24 40 DOWN 0/0
Vl200 101 0 192.168.200.1/24 40 DR 0/0
Vl199 101 0 192.168.199.1/24 40 DOWN 0/0
Vl193 101 0 192.168.193.1/24 40 DOWN 0/0
Vl190 101 0 192.168.190.1/24 40 DR 0/0
Vl188 101 0 192.168.188.1/24 40 DOWN 0/0
Vl187 101 0 192.168.187.1/24 40 DR 0/0
Vl158 101 0 192.168.158.1/24 40 DOWN 0/0
Vl157 101 0 192.168.157.1/24 40 DR 0/0
Vl156 101 0 192.168.156.1/24 40 DR 0/0
Vl154 101 0 192.168.154.1/24 40 DR 0/0
Vl152 101 0 192.168.152.1/24 40 DR 0/0
Vl119 101 0 192.168.119.1/24 40 DR 0/0
Vl112 101 0 192.168.112.1/24 40 DR 0/0
Vl111 101 0 192.168.111.1/24 40 DOWN 0/0
Vl110 101 0 192.168.110.1/24 40 DR 0/0
Vl109 101 0 192.168.109.1/24 40 DR 0/0
Vl101 101 0 192.168.101.1/24 40 DR 0/0
Vl96 101 0 192.168.96.1/24 40 DR 0/0
Vl95 101 0 192.168.95.1/24 40 DR 0/0
Vl89 101 0 192.168.89.1/24 40 DOWN 0/0
Vl88 101 0 192.168.88.1/24 40 DOWN 0/0
Vl84 101 0 192.168.84.1/24 40 DR 0/0
Vl83 101 0 192.168.83.1/24 40 DR 0/0
Vl82 101 0 192.168.82.1/24 40 DR 0/0
Vl81 101 0 192.168.81.1/24 40 DR 0/0
Vl80 101 0 192.168.80.1/24 40 DR 0/0
Vl79 101 0 192.168.79.1/24 40 DR 0/0
Vl76 101 0 192.168.76.1/24 40 DR 0/0
Vl75 101 0 192.168.75.1/24 40 DR 0/0
Vl74 101 0 192.168.74.1/24 40 DR 0/0
Vl73 101 0 192.168.73.1/24 40 DR 0/0
Vl72 101 0 192.168.72.1/24 40 DR 0/0
Vl70 101 0 192.168.70.1/24 40 DOWN 0/0
Vl69 101 0 192.168.69.1/24 40 DR 0/0
Vl68 101 0 192.168.68.1/24 40 DR 0/0
Vl67 101 0 192.168.67.1/24 40 DR 0/0
Vl65 101 0 192.168.65.1/24 40 DOWN 0/0
Vl64 101 0 192.168.64.1/24 40 DR 0/0
Vl61 101 0 192.168.61.1/24 40 DR 0/0
Vl60 101 0 192.168.60.1/24 40 DR 0/0
Vl59 101 0 192.168.59.1/24 40 DR 0/0
Vl58 101 0 192.168.58.1/24 40 DR 0/0
Vl57 101 0 192.168.57.1/24 40 DR 0/0
Vl56 101 0 192.168.56.1/24 40 DR 0/0
Vl55 101 0 192.168.55.1/24 40 DR 0/0
Vl54 101 0 192.168.54.1/24 40 DR 0/0
Vl45 101 0 192.168.45.1/24 40 DR 0/0
Vl44 101 0 192.168.44.1/24 40 DR 0/0
Vl19 101 0 192.168.19.1/24 40 DR 0/0
Vl12 101 0 192.168.12.1/24 40 DR 0/0
Vl11 101 0 192.168.11.1/24 40 DR 0/0
Vl9 101 0 192.168.9.1/24 40 DR 0/0
Vl7 101 0 192.168.7.1/24 40 DR 0/0
Vl6 101 0 192.168.6.1/24 40 DOWN 0/0
Vl5 101 0 192.168.5.1/24 40 DROTH 2/2
Vl1 101 0 192.168.1.1/24 40 DR 0/0
Vl502 101 0 10.255.255.33/30 40 BDR 1/1
Vl501 101 0 10.255.255.29/30 40 BDR 1/1
Vl811 101 1 10.255.255.133/30 40 BDR 1/1
Vl809 101 1 10.255.255.125/30 40 DR 1/1
Vl808 101 1 10.255.255.121/30 40 DR 1/1
Vl807 101 1 10.255.255.117/30 40 DR 1/1
Vl810 101 1 10.255.255.129/30 40 BDR 1/1
Vl10 101 1 10.0.2.1/23 40 DR 0/0
Vl759 101 1 172.17.14.1/24 40 DR 0/0
Vl758 101 1 172.17.13.1/24 40 DR 0/0
Vl757 101 1 172.17.12.1/24 40 DR 0/0
Vl756 101 1 172.17.11.1/24 40 DR 0/0
Vl755 101 1 172.17.10.1/24 40 DR 0/0
Vl256 101 1 192.168.215.65/26 40 DR 0/0
Vl611 101 1 172.22.11.1/24 40 DR 0/0
Vl721 101 1 172.21.0.1/21 40 DR 0/0
Vl266 101 1 172.19.0.1/21 40 DR 0/0
Vl754 101 1 172.17.8.1/23 40 DR 0/0
Vl753 101 1 172.17.6.1/23 40 DR 0/0
Vl752 101 1 172.17.4.1/23 40 DR 0/0
Vl751 101 1 172.17.2.1/23 40 DR 0/0
Vl750 101 1 172.17.0.1/23 40 DR 0/0

Site 2

Interface PID Area IP Address/Mask Cost State Nbrs F/C
Vl804 101 0 10.255.255.98/30 40 DR 0/0
Vl42 101 0 192.168.42.1/24 40 DR 0/0
Vl501 101 0 10.255.255.30/30 40 DR 1/1
Vl232 101 2 10.2.32.1/24 40 DR 0/0
Vl43 101 2 192.168.43.1/24 40 DR 0/0
Vl41 101 2 192.168.41.1/24 40 DOWN 0/0
Vl40 101 2 192.168.40.1/24 40 DR 0/0
Vl39 101 2 192.168.39.1/24 40 DR 0/0
Vl38 101 2 192.168.38.1/24 40 DR 0/0
Vl37 101 2 192.168.37.1/24 40 DR 0/0
Vl35 101 2 192.168.35.1/24 40 DOWN 0/0
Vl34 101 2 192.168.34.1/24 40 DR 0/0
Vl33 101 2 192.168.33.1/24 40 DR 0/0
Vl32 101 2 192.168.32.1/24 40 DR 0/0
Vl30 101 2 192.168.30.1/24 40 DR 0/0
Vl29 101 2 192.168.29.1/24 40 DR 0/0
Vl28 101 2 192.168.28.1/24 40 DR 0/0
Vl27 101 2 192.168.27.1/24 40 DR 0/0
Vl26 101 2 192.168.26.1/24 40 DR 0/0
Vl25 101 2 192.168.25.1/24 40 DR 0/0
Vl24 101 2 192.168.24.1/24 40 DR 0/0
Vl23 101 2 192.168.23.1/24 40 DR 0/0
Vl22 101 2 192.168.22.1/24 40 DR 0/0
Vl21 101 2 192.168.21.1/24 40 DR 0/0
Vl20 101 2 192.168.20.1/24 40 DR 0/0
Vl477 101 2 172.20.8.65/26 40 DR 0/0

Site 3

Interface PID Area IP Address/Mask Cost State Nbrs F/C
Vl146 101 0 192.168.146.1/24 40 DR 0/0
Vl130 101 0 192.168.130.1/24 40 DOWN 0/0
Vl503 101 0 10.255.255.37/30 50 BDR 1/1
Vl502 101 0 10.255.255.34/30 40 DR 1/1
Vl360 101 3 10.3.60.1/24 40 DR 0/0
Vl356 101 3 10.3.56.1/23 40 DR 0/0
Vl352 101 3 10.3.52.1/23 40 DR 0/0
Vl348 101 3 10.3.48.1/23 40 DR 0/0
Vl343 101 3 10.3.43.1/24 40 DR 0/0
Vl342 101 3 10.3.42.1/24 40 DR 0/0
Vl340 101 3 10.3.40.1/24 40 DR 0/0
Vl332 101 3 10.3.32.1/23 40 DR 0/0
Vl324 101 3 10.3.24.1/23 40 DR 0/0
Vl316 101 3 10.3.16.1/23 40 DR 0/0
Vl308 101 3 10.3.8.1/24 40 DR 0/0
Vl304 101 3 10.3.4.1/24 40 DR 0/0
Vl302 101 3 10.3.2.1/24 40 DR 0/0
Vl301 101 3 10.3.1.1/24 40 DR 0/0
Vl300 101 3 10.3.0.1/24 40 DR 0/0
Vl476 101 3 172.20.10.65/26 40 DR 0/0
Vl631 101 3 172.20.10.1/26 40 DR 0/0
Vl148 101 3 192.168.148.1/24 40 DR 0/0
Vl145 101 3 192.168.145.1/24 40 DR 0/0
Vl144 101 3 192.168.144.1/24 40 DR 0/0
Vl142 101 3 192.168.142.1/24 40 DR 0/0
Vl140 101 3 192.168.140.1/24 40 DR 0/0
Vl139 101 3 192.168.139.1/24 40 DR 0/0
Vl138 101 3 192.168.138.1/24 40 DR 0/0
Vl137 101 3 192.168.137.1/24 40 DR 0/0
Vl136 101 3 192.168.136.1/24 40 DR 0/0
Vl135 101 3 192.168.135.1/24 40 DR 0/0
Vl134 101 3 192.168.134.1/24 40 DR 0/0
Vl133 101 3 192.168.133.1/24 40 DR 0/0
Vl132 101 3 192.168.132.1/24 40 DR 0/0
Vl131 101 3 192.168.131.1/24 40 DR 0/0

Site 4

Interface PID Area IP Address/Mask Cost State Nbrs F/C
Vl804 101 0 10.255.255.97/30 40 DR 0/0
Vl815 101 0 10.255.255.105/30 40 DR 0/0
Vl178 101 0 192.168.178.1/24 40 DR 0/0
Vl159 101 0 192.168.159.1/24 40 DR 0/0
Vl503 101 0 10.255.255.38/30 40 DR 1/1
Vl476 101 4 10.4.76.1/24 40 DR 0/0
Vl180 101 4 192.168.180.1/24 40 DR 0/0
Vl179 101 4 192.168.179.1/24 40 DR 0/0
Vl177 101 4 192.168.177.1/24 40 DOWN 0/0
Vl176 101 4 192.168.176.1/24 40 DR 0/0
Vl175 101 4 192.168.175.1/24 40 DOWN 0/0
Vl174 101 4 192.168.174.1/24 40 DR 0/0
Vl173 101 4 192.168.173.1/24 40 DR 0/0
Vl172 101 4 192.168.172.1/24 40 DR 0/0
Vl171 101 4 192.168.171.1/24 40 DR 0/0
Vl170 101 4 192.168.170.1/24 40 DR 0/0
Vl169 101 4 192.168.169.1/24 40 DR 0/0
Vl168 101 4 192.168.168.1/24 40 DOWN 0/0
Vl167 101 4 192.168.167.1/24 40 DR 0/0
Vl166 101 4 192.168.166.1/24 40 DR 0/0
Vl165 101 4 192.168.165.1/24 40 DR 0/0
Vl164 101 4 192.168.164.1/24 40 DR 0/0
Vl163 101 4 192.168.163.1/24 40 DR 0/0
Vl162 101 4 192.168.162.1/24 40 DR 0/0
Vl161 101 4 192.168.161.1/24 40 DR 0/0
Vl160 101 4 192.168.160.1/24 40 DOWN 0/0
Vl475 101 4 172.20.9.65/26 40 DR 0/0
Vl661 101 4 172.20.9.1/26 40 DR 0/0
Vl467 101 4 10.254.67.1/26 40 DR 0/0

Reza Sharifi · ‎06-22-2023

Can you also post the output of "sh spanning vlan 804" from both r2 and r4?

mccoyb_scc · ‎06-22-2023

Site 4

MST0
Spanning tree enabled protocol mstp
Root ID Priority 4096
Address 40f0.78aa.9700
Cost 0
Port 410 (HundredGigE2/0/26)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768 (priority 32768 sys-id-ext 0)
Address 549f.c6bd.0e60
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Twe1/0/3 Desg FWD 2000 128.195 P2p
Twe1/0/4 Desg FWD 2000 128.196 P2p
Twe1/0/5 Desg FWD 2000 128.197 P2p
Twe1/0/17 Desg FWD 2000 128.209 P2p
Twe1/0/21 Desg FWD 2000 128.213 P2p
Hu1/0/25 Altn BLK 500 128.217 P2p
Twe2/0/1 Desg FWD 2000 128.385 P2p
Twe2/0/2 Desg FWD 2000 128.386 P2p
Twe2/0/3 Desg FWD 2000 128.387 P2p
Twe2/0/17 Desg FWD 2000 128.401 P2p
Twe2/0/19 Desg FWD 20000 128.403 P2p
Twe2/0/21 Desg FWD 20000 128.405 P2p

Site 2

MST0
Spanning tree enabled protocol mstp
Root ID Priority 4096
Address 40f0.78aa.9700
Cost 0
Port 218 (HundredGigE1/0/26)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768 (priority 32768 sys-id-ext 0)
Address 549f.c6bd.0dc0
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Twe1/0/2 Desg FWD 20000 128.194 P2p
Twe1/0/8 Desg FWD 20000 128.200 P2p
Twe1/0/12 Desg FWD 2000 128.204 P2p
Twe2/0/1 Desg FWD 2000 128.385 P2p
Twe2/0/2 Desg FWD 2000 128.386 P2p
Twe2/0/3 Desg FWD 2000 128.387 P2p
Hu2/0/26 Desg FWD 500 128.410 P2p

Reza Sharifi · ‎06-22-2023

Hu1/0/25 Altn BLK 500 128.217 P2p

So, 1/0/25 is blocked and that is why there is no peering between 2 and 4.

As I said in my first post, there is no need to extend all these vlans across the link between 2 and 4. You only need one transit vlan for peering and that is 804. This way you keep the layer-3 boundaries separate and no link will be blocked as long as OSPF is configured correctly. The problem with your design is that you are mixing L2 and L3 together.

HTH

mccoyb_scc · ‎06-22-2023

I understand about the design and it is not what I wanted when designing, but mixing is unavoidable. I also understand that STP has the link blocked, that is normal as it is our longest fiber run. The problem is that even when we simulate a fiber break the L3 does not come up. Also, I should be able to ping either VLAN interface at any time. I cannot do this on site 4 VLAN 804 unless I shut site 2 VLAN 804. This is new. This design has been working for years. First with 6509 and 4500 and now with the 9500, until recently.

MHM Cisco World · ‎06-22-2023

All link connect to DR there is no BDR or DRother'

I think issue here is you running opsf in ehternet link which by defualt broadcast but actually your topolgy is hub to spoke or NBMA.

Show ip ospf neighbor brief<<- share this to check if I am right or not

Reza Sharifi · ‎06-22-2023

OV-Core-9500#show ip ospf interface
Vlan804 is up, line protocol is up 
  Internet Address 10.255.255.98/30, Interface ID 111, Area 0
  Attached via Network Statement
  Process ID 101, Router ID 2.1.1.1, Network Type BROADCAST, Cost: 40
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           40        no          no            Base

mccoyb_scc · ‎06-22-2023

I am unsure why you posted this. I can ping 10.255.255.98. Site 4 also shows the ospf int up and I cannot ping it.

ALT-Core-9500#sh ip os int
Vlan804 is up, line protocol is up
Internet Address 10.255.255.97/30, Interface ID 117, Area 0
Attached via Network Statement
Process ID 101, Router ID 4.1.1.1, Network Type BROADCAST, Cost: 40
Topology-MTID Cost Disabled Shutdown Topology Name
0 40 no no Base

mccoyb_scc · ‎06-22-2023

We are currently revising the DR site and changing locations. It is new and all connections not yet established. AT&T is not the fastest mover. DR is 5.1.1.1

Neighbor ID Pri State Dead Time Address Interface
5.1.1.1 1 FULL/BDR 00:00:36 10.255.255.46 TwentyFiveGigE1/0/13
1.1.1.10 1 FULL/BDR 00:00:34 10.255.255.78 Vlan515
192.168.5.7 1 FULL/BDR 00:00:36 192.168.5.7 Vlan5
192.168.5.9 1 FULL/DR 00:00:32 192.168.5.9 Vlan5
3.1.1.1 1 FULL/DR 00:00:31 10.255.255.34 Vlan502
2.1.1.1 1 FULL/DR 00:00:35 10.255.255.30 Vlan501
1.1.1.6 1 FULL/DR 00:00:33 10.255.255.134 Vlan811
1.1.1.7 1 FULL/BDR 00:00:35 10.255.255.126 Vlan809
1.1.1.4 1 FULL/BDR 00:00:34 10.255.255.122 Vlan808
1.1.1.8 1 FULL/BDR 00:00:33 10.255.255.118 Vlan807
1.1.1.5 1 FULL/DR 00:00:33 10.255.255.130 Vlan810

MHM Cisco World · ‎06-22-2023

@Reza Sharifi I know he use medium broadcast but the topology not broadcast.

@mccoyb_scc you have four SW i.e. four ospf peer

Check each peer for specifc vlan for example vlan 809' you must see DR/BDR and two DRother

1.1.1.7 1 FULL/BDR 00:00:35 10.255.255.126 Vlan809

Unless you use transit vlan to connect four site' which I dont think so' because you mention that there stp and it work and it blk one link.

mccoyb_scc · ‎06-22-2023

Sorry. The following are not part of the ring and are distributed routing on the main campus.

1.1.1.6 1 FULL/DR 00:00:39 10.255.255.134 Vlan811
1.1.1.7 1 FULL/BDR 00:00:36 10.255.255.126 Vlan809
1.1.1.4 1 FULL/BDR 00:00:36 10.255.255.122 Vlan808
1.1.1.8 1 FULL/BDR 00:00:37 10.255.255.118 Vlan807
1.1.1.5 1 FULL/DR 00:00:36 10.255.255.130 Vlan810

mccoyb_scc · ‎06-22-2023

Thank everyone for their help. We have opened a TAC case. I will report the resolution once it is reached.

MHM Cisco World · ‎06-22-2023

Meanwhile I will try do some lab and check again all info. You share.

Have a nice day

MHM

MHM Cisco World · ‎06-24-2023