I have gone back and read again all of the posts in this extensive discussion. And I realize that many of my posts have addressed this network in terms of how I would want it to process, and have not adequately recognized how it does currently process. So let me say some things about the current environment and then about what I would want it to be like.
Let me also say that part of the confusion has been that some things have shifted as the discussion has progressed. Originally OSPF was using vlan 11 and subnet 10.2.2.0 and in later posts OSPF is using vlan 15 and 10.2.52.0. Not sure what else might have shifted, and believe that ultimately the changes are not significant.
I believe that it might help to discuss this network design as having 2 major parts. One part is the routing logic for inside the network and the other part is the routing logic for getting outside of the network.
Routing inside of the network uses the switches (for vlan assignments and forwarding at layer 2) and the firewalls for layer 3 logic (and the routers have no role in any of this). The firewalls have knowledge of all inside subnets, have responsibility for routing between inside subnets, have responsibility for forwarding from inside subnets to outside. We have not seen any details about this but it is logical to assume that the firewalls set their default gateway for access outside to use the GLBP virtual address.
Routing for getting outside of the network is on the routers (and the firewalls and the core switches have no role in this). In the environment at the beginning of the discussion each router had its own static default route with its BGP peer as the next hop. The forwarding logic of GLBP would decide which router received traffic from inside going outside and the router that received that traffic would forward outside using its static default route. So the failover mechanism is GLBP and static default route.
It seems to me that this architecture works ok as long as all its parts are working. But I am concerned about what happens when something is not working as intended.
1) The logic that we have seen for GLBP will choose one of the routers to be the active router. We have not seen any logic that would recognize a problem on the active router (especially loss of connectivity to its BGP peer) and shift the active router to the other router. Does that logic exist?
2) As I discussed in a previous post there are issues with a static default route, especially the possibility that you could lose connectivity to the BGP peer but the interface not go to the down state. In that situation the default route would continue to be active, but would be trying to forward traffic to a next hop that is not reachable.
So my opinion is that is some circumstances the routers might failover (as apparently was tested at one point) if an interface was shut down or the router taken out of service. But in some other circumstances failover might not work as expected.
In terms of how I might want to see this network operate I would prefer that the failover mechanism be based on BGP rather than GLBP and static default routes. Assuming that each ISP does advertise a default route then each router would learn a default route and if there was a problem with one of the ISP then its default route would be withdrawn and the routers would realize that they need to use the other router to get outside.
In several respects I believe that the network configuration is more complex than it needs to be and would suggest some simplification. It appears that both ISP are advertising the full BGP set of Internet routes. I do not see a reason why this organization needs to receive 2 sets of complete routes. Receiving the full Internet set of routes makes sense for an organization that wants to use both ISP in a load sharing operation where some routes are best through one ISP while other routes are best through the other ISP. But clearly this organization is operating in an active BGP/backup BGP where all traffic is sent through one ISP and the second ISP is used only for failover. Requesting each ISP to advertise just a default route would significantly reduce the processing overhead on each router and would achieve the primary/backup relationship more efficiently.
I also think the IBGP is more complex than it needs to be. For example the use of peer groups in the IBGP would be appropriate when there are more than 2 IBGP routers. But when there are only 2 I do not see much benefit from peer groups. Another example of unnecessary complexity is the use of the Loopback interface for the IBGP peer. Using loopback interface addresses makes good sense when there are more than one path to reach the peer (if the primary path fails there could be a second path to reach the IBGP peer). On these routers there are 2 vlans that connect them. But clearly vlan 14 is for only Inside traffic and vlan 11/15 is for outside traffic. If there is no alternate path then there is no advantage in using loopback interfaces for IBGP peering.
And using the vlan 11/15 interface address as the IBGP peering address would remove the need for OSPF. I have been quite slow to recognize that in the current environment that OSPF is indeed required for IBGP - because it is OSPF that lets one router know how to reach the IBGP peer address. If the IBGP peering used the vlan interface address then there is no need for OSPF.
HTH
Rick
