Extend subnet from /24 to 23 - not as easy as I hoped

[RRatBB]

All IPs, AS numbers, OSPF areas, etc., have been sanitized for this post. None are currently in production.

- I have a subnet at a remote network that I need to extend from a /24 to a /23.
- We have two circuits at this site: a primary MPLS going to our Cisco router, and a secondary broadband going to a firewall for backup VPN tunnel.
- Networks are advertised on the router using BGP and OSPF

My plan was to:

1. Write mem on the router
2. Set reload in 10 just in case something happens
3. Change router interface gi0/0.1 from:
ip address 10.10.10.1 255.255.255.0
to ip address 10.10.10.1 255.255.254.0
4. Change mask on OSPF advertisement from 0.0.0.255 to 0.0.1.255
6. Change mask on BGP advertisement from 255.255.255.0 to 255.255.254.0
7. Move on to changing network settings on the firewall interface, switch management IPs, DHCP, etc.

Well, I got through steps 1 and 2 okay.

But as soon as I changed the ip subnet ((config)#ip address 10.10.10.1 255.255.254.0) on the Gi0/0.1 interface, I lost connectivity to the router, and after a few seconds our VPN backup tunnel activated. After the VPN came up, I could ping the router, but not SSH.

After 10 minutes the router reloaded and things went back to normal shortly thereafter.

So, I'm wondering where I went wrong?

- I am wondering if because I didn't change the OSPF and BGP advertisements first, this caused the VPN tunnel to come up, killed my connection to the router, and then I couldn't SSH into it because of some VPN rule on the firewall? Should I change the OSPF and BGP masks before I change the interface mask?

- Or if changing the mask on the Gi0/0.1 interface broke the routing in some other way? The router "LAN" interface is connected to the firewall. Maybe I should have changed the firewall interface first?

What is the recommended order of doing this? Google/YouTube makes it sound easy, but obviously I am missing something.

Thanks

[David Ruess]

Hello,

I would definitely change the advertisements on OSPF and BGP first. Doing so should lose you connectivity and if that interface IP/netowrks doesnt exist it just wont advertise.

Secondly do your neighborships for OSPF and BGP come back up. You said you were able to ping it. Where? From the directly connected router or your workstation? 

I would also check the logs to see if SSH was denied or even hitting your device after the change. After making remote changes make sure local devices also are changed to match so the routing neighbors come up.

If you provided configurations and a diagram, we may be better able to assist you.

-David

[ksolie]

I don't know if there is any real way you could have avoided this. 

Especially if you were SSH to that address, or relied on the OSPF/BGP advertisement to get to the address you used for SSH. I usually use a loopback for management. But again if it relies on that route you changed to reach that loopback, no way to avoid the outage. 

You may have made a mistake on the OSPF or BGP. Do a show ip bgp neighbor x.x.x.x advertise-routes and a show ip ospf interface. I think those commands are close doing it my memory and no ?. hehehe

[Joseph W. Doherty]

I don't know if there is any real way you could have avoided this. 

Especially if you were SSH to that address, or relied on the OSPF/BGP advertisement to get to the address you used for SSH. I usually use a loopback for management. But again if it relies on that route you changed to reach that loopback, no way to avoid the outage. 

I found it especially concerning there's a backup path, which one would assume should have allowed continuous or a reconnection to the router.  So, without much more information, cannot say, with much confidence, why the described happen.

Also, in the past, have found it difficult to sometimes ascertain the cause if OP will not provide requested information. Don't know if that may be the case here, but first paragraph of OP indicates it may be.

[ksolie]

also for SSH, make sure you don't have an ACL applied to it with the wrong subnet/host address. 

Your order is correct. Change the interface first, then OSPF, then BGP. BGP will not advertise a route it doesn't see match the MASK or the network command. 

[Joseph W. Doherty]

Without much more detail, cannot say much on why you had the problems you describe.  I presume the /24 to /23 would be an edge network, so I wouldn't think you would need to transit it, but again, have no idea what the overall OSPF and/or BGP topology/configuration are.

If an edge network, any particular need why the subinterface couldn't have been shut while all necessary configuration changes made?

BTW, something seen on Google/YouTube are not necessarily guarantees of quality, or being foolproof.  That's not to say what they present is totally wrong, just they may overlook a "gotcha" or two.

[RRatBB]

Thanks to all for the feedback.  Trying to put together some more information. I'm not great with Visio, as you can tell. Also trying to sanitize configs.  Anyway, see attached.  But a couple of helpful things have come out of this conversation that I am going to try tomorrow:  1) Try doing this from a different interface so that I am not working on the interface I am connected to.  Seems pretty basic.  2) Make sure I am not blocking SSH when I fail over to backup path.  I already considered this, just haven't had time to look at it yet.  Will try to do so today.  Anyway, I appreciate the feedback. I'll post my results tomorrow. 

[RRatBB]

Dang it ... always happens.  Ignore the AS number for BGP in my diagram.  That's a typo.  But they *do* match on my actual configs. Haven't made any changes there and it's all currently working.

[RRatBB]

I was able to make the changes without kicking myself off the device ... so some progress made!

However, when I changed the /24 subnet to a /23 subnet on all three items (interface, ospf advertisement, and bgp advertisement), it still failed over to our backup vpn tunnel and stayed there until I reverted back to /24 on everything. When I moved everything *back* to the /24 subnet, it failed back across our normal circuit almost immediately.

#show ip bgp
#show ip route
#show bgp

These look the same before and after, so maybe the problem is not on this router but upstream somewhere. We don't have any prefix-lists on our core, and we have one route-map statement:

route-map BGP-IN, permit, sequence 20
Match clauses:
ip address (access-lists): 20

And access-list 20 has this statement:
access-list 20 permit 10.10.0.0 0.0.255.255

So <shrug>, I dunno.

Here is what I found:

BEFORE CHANGES (everything working):

interface GigabitEthernet0/0.1
description Data VLAN
encapsulation dot1Q 1 native
ip address 10.10.10.1 255.255.255.0
ip flow ingress
ip flow egress

router ospf 1
router-id 10.10.10.1
redistribute bgp 65005 subnets
network 10.10.10.0 0.0.0.255 area 1
!
router bgp 65005
bgp log-neighbor-changes
network 10.10.10.0 mask 255.255.255.0
neighbor 100.75.1.99 remote-as 209
neighbor 100.75.1.99 allowas-in
!
BRANCH01#
BRANCH01#
BRANCH01#
BRANCH01#sh ip bgp 10.10.10.0
BGP routing table entry for 10.10.10.0/24, version 69
Paths: (1 available, best #1, table default)
Advertised to update-groups:
1
Refresh Epoch 1
Local
0.0.0.0 from 0.0.0.0 (100.75.1.99)
Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best
rx pathid: 0, tx pathid: 0x0
BRANCH01#
BRANCH01#sh ip route 10.10.10.0
Routing entry for 10.10.10.0/24
Known via "connected", distance 0, metric 0 (connected, via interface)
Redistributing via ospf 1
Advertised by bgp 65005
Routing Descriptor Blocks:
* directly connected, via GigabitEthernet0/0.1
Route metric is 0, traffic share count is 1
BRANCH01#
BRANCH01#show bgp 10.10.10.0
BGP routing table entry for 10.10.10.0/24, version 69
Paths: (1 available, best #1, table default)
Advertised to update-groups:
1
Refresh Epoch 1
Local
0.0.0.0 from 0.0.0.0 (100.75.1.99)
Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best
rx pathid: 0, tx pathid: 0x0
BRANCH01#

===============================================
AFTER CHANGES (fails over to backup tunnel):

!
interface GigabitEthernet0/0.1
description Data VLAN
encapsulation dot1Q 1 native
ip address 10.10.10.1 255.255.254.0
ip flow ingress
ip flow egress
!
!
router ospf 1
router-id 10.10.10.1
redistribute bgp 65005 subnets
network 10.10.10.0 0.0.1.255 area 1
!
router bgp 65005
bgp log-neighbor-changes
network 10.10.10.0 mask 255.255.254.0
neighbor 100.75.1.99 remote-as 209
neighbor 100.75.1.99 allowas-in
!
BRANCH01#show ip bgp 10.10.10.0
BGP routing table entry for 10.10.10.0/23, version 67
Paths: (1 available, best #1, table default)
Advertised to update-groups:
1
Refresh Epoch 1
Local
0.0.0.0 from 0.0.0.0 (100.75.1.99)
Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best
rx pathid: 0, tx pathid: 0x0
BRANCH01#

BRANCH01#
BRANCH01#show ip route 10.10.10.0
Routing entry for 10.10.10.0/23
Known via "connected", distance 0, metric 0 (connected, via interface)
Redistributing via ospf 1
Advertised by bgp 65005
Routing Descriptor Blocks:
* directly connected, via GigabitEthernet0/0.1
Route metric is 0, traffic share count is 1
BRANCH01#

BRANCH01#show bgp 10.10.10.0
BGP routing table entry for 10.10.10.0/23, version 67
Paths: (1 available, best #1, table default)
Advertised to update-groups:
1
Refresh Epoch 1
Local
0.0.0.0 from 0.0.0.0 (100.75.1.99)
Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best
rx pathid: 0, tx pathid: 0x0
BRANCH01#

[Joseph W. Doherty]

Did you shut the subinterface before making changes?

[RRatBB]

I did not shutdown the subinterface before making changes. I will give that a try, but it is acting as a gateway for that vlan so I will have to do it after hours. Thanks

[MHM Cisco World]

You have vpn tunnel' what is tunnel type? Policy or route based vpn??

MHM

[RRatBB]

Route-based.

Thanks

[MHM Cisco World]

You have vti and mpls' 

What protocol you run for vti and mpls ?

MHM

[RRatBB]

BGP and OSPF

router ospf 1
router-id 10.10.10.1
redistribute bgp 65005 subnets
network 10.10.10.0 0.0.0.255 area 1
network 10.10.10.0 0.0.1.255 area 1
!
router bgp 65005
bgp log-neighbor-changes
network 10.10.10.0 mask 255.255.254.0
network 10.10.10.0 mask 255.255.255.0
neighbor 100.75.1.99 remote-as 209
neighbor 100.75.1.99 allowas-in

I think maybe something might be messed up with OSPF. When everything is set to the /24 subnet, I see this:

BRANCH01#sh ip ospf neigh

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 1 FULL/DR 00:00:36 10.10.10.2 GigabitEthernet0/0.1
BRANCH01#

When I set everything to the /23 subnet, I see this:

BRANCH01#sh ip ospf neigh
BRANCH01#