Curious about correct setup of ACL / PBR or Static Routing

[TheGoob]

Hello

I have a working system coming from another Thread which was solved but now I am curious if how I got it all to work was correct or a forceful band-aid. 

https://community.cisco.com/t5/routing/can-not-get-internet-access-or-nat-translations-on-my-isrc1111/td-p/5342191

Now, some of that has changed as I was figuring out other things, but the result is to be the same; 7 vlans, 6 going towards ISP1 and 1 going towards ISP2 but all can communicate together through the SG350XG Switch in which they connect to without having to touch each Router then route back to the Switch [inter-vlan]. Not as simple as it seems as there are two initial routes for them to connect to their own ISP's. 

What I have done is, vlan 2- 7 use the static route 0.0.0.0 0.0.0.0 10.0.1.1 to reach ISP1 and vlan 8 I can not utilize it's own static route to confuse the default static route so I created a PBR to reach ISP2. Works fine!! Until I realized they would all communicate but have to route to their respective routers then route back to communicate. This is important to me because my Network is 10G, 10G Nics, 10G Switch etc and so if/when I transfer from vlan 8 to let's say 5, it would slow down to 1G to route through the ISR. So I created the ACL to block vlan 2-7 but allow 8; which works. With default next-hop and PBR and ACL, is my setup legit?

switchbf585b
!
vlan database
vlan 2-8 
exit
ip dhcp server 
ip dhcp pool network 4.0
address low 192.168.4.2 high 192.168.4.100 255.255.255.0 
exit
ip dhcp pool network 5.0
address low 192.168.5.2 high 192.168.5.254 255.255.255.0 
dns-server 1.1.1.1
exit
ip dhcp pool network 6.0
address low 192.168.6.2 high 192.168.6.254 255.255.255.0 
exit
ip dhcp pool network fhc
address low 192.168.2.2 high 192.168.2.154 255.255.255.0 
exit
ip dhcp pool network ceyea
address low 192.168.3.6 high 192.168.3.254 255.255.255.0 
exit
ip dhcp pool network fbeye
address low 192.168.1.2 high 192.168.1.254 255.255.255.0 
exit
ip dhcp pool network starlink
address low 192.168.7.1 high 192.168.7.254 255.255.255.0 
dns-server 8.8.8.8
exit
bonjour interface range oob
ip access-list extended SL
deny ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255 ace-priority 5
permit ip 192.168.7.0 0.0.0.255 any ace-priority 10
exit
route-map sl 1 
 match ip address access-list SL 
 set ip next-hop 10.0.2.1 
exit
ip name-server  8.8.8.8 1.1.1.1 192.168.3.5
!
interface vlan 2
 name fbeye 
 ip address 192.168.1.1 255.255.255.0 
!
interface vlan 3
 name fhc 
 ip address 192.168.2.1 255.255.255.0 
!
interface vlan 4
 name ceyea 
 ip address 192.168.3.1 255.255.255.0 
!
interface vlan 5
 name 4.0 
 ip address 192.168.4.1 255.255.255.0 
!
interface vlan 6
 name 6.0 
 ip address 192.168.6.1 255.255.255.0 
!
interface vlan 7
 name home 
 ip address 192.168.5.1 255.255.255.0 
!
interface vlan 8
 name starlink 
 ip address 192.168.7.1 255.255.255.0 
 ip policy route-map sl 
!
interface TenGigabitEthernet1/0/1
 ip address 10.0.2.2 255.255.255.0 
 no switchport 
 switchport access vlan none 
 switchport trunk native vlan none 
!
interface TenGigabitEthernet1/0/2
 switchport access vlan 2 
 switchport trunk native vlan 2 
!
interface TenGigabitEthernet1/0/3
 switchport access vlan 3 
 switchport trunk native vlan 3 
!
interface TenGigabitEthernet1/0/4
 switchport access vlan 4 
 switchport trunk native vlan 4 
!
interface TenGigabitEthernet1/0/5
 switchport access vlan 5 
 switchport trunk native vlan 5 
!
interface TenGigabitEthernet1/0/6
 switchport access vlan 6 
 switchport trunk native vlan 6 
!
interface TenGigabitEthernet1/0/7
 switchport access vlan 7 
 switchport trunk native vlan 7 
!
interface TenGigabitEthernet1/0/8
 switchport access vlan 8 
 switchport trunk native vlan 8 
!
interface TenGigabitEthernet1/0/9
 ip address 10.0.0.2 255.255.255.0 
 no switchport 
 switchport access vlan none 
 switchport trunk native vlan none 
!
interface TenGigabitEthernet1/0/10
 switchport access vlan 8 
 switchport trunk native vlan 8 
 no macro auto smartport 
!
interface oob
 ip address 192.168.10.254 255.255.255.0 
 no ip address dhcp 
!
exit
ip default-gateway 10.0.0.1 

[Enes Simnica · Accepted solution]

gDay @TheGoob Ur setup is actually solid. Cause using a default route for VLANs 2–7 and PBR for VLAN 8 is a valid, clean way to handle dual-ISP routing. The ACL to keep inter-VLAN traffic local on the SG350XG is also smart, since it avoids dropping to 1 G through the routers. 

long story short G, setup is clean. Don't second-guess it!

cool profile picture btw AHAHHAHAHAAH sick!

-Enes

[elwin-berrar · Accepted solution]

Agree with @enes, setup looks clean.
Default route for VLANs 2–7 and PBR for VLAN 8 is exactly how I’d handle dual ISPs here.
Keeping inter-VLAN traffic local on the SG350XG makes total sense with 10 G links.

[TheGoob]

Well awesome! Thank you both for the reassurance. I am happy

[TheGoob]

Hello

I will leave this thread as solved but I seem to have run into an issue when migrating over to a 3850 Switch from the working SG350XG Switch.. I have the vlans, acl's, pbr all set the exact same but it appears everything is back to routing through the routers then back to the Switch instead of just routing/staying local on the Switch to utilize the 10G xfer rates among vlans. Here is the "new" / 3850 Switch config.. If anyone happens to see something I had missed.

!
version 16.12
!
ip routing
!
ip dhcp excluded-address 192.168.1.0 192.168.1.1
!
ip dhcp pool fbeye
 network 192.168.1.0 255.255.255.0
 default-router 192.168.1.1 
 dns-server 1.1.1.1 
 lease infinite
!
ip dhcp pool fhc
 network 192.168.2.0 255.255.255.0
 default-router 192.168.2.1 
 dns-server 1.1.1.1 
 lease infinite
!
ip dhcp pool ceyea
 network 192.168.3.0 255.255.255.0
 default-router 192.168.3.1 
 dns-server 1.1.1.1 
 lease infinite
!
ip dhcp pool 4.0
 network 192.168.4.0 255.255.255.0
 default-router 192.168.4.1 
 dns-server 1.1.1.1 
 lease infinite
!
ip dhcp pool 5.0
 network 192.168.5.0 255.255.255.0
 default-router 192.168.5.1 
 dns-server 1.1.1.1 
 lease infinite
!
ip dhcp pool 6.0
 network 192.168.6.0 255.255.255.0
 default-router 192.168.6.1 
 dns-server 1.1.1.1 
 lease infinite
!
ip dhcp pool starlink
 network 192.168.7.0 255.255.255.0
 default-router 192.168.7.1 
 dns-server 1.1.1.1 
 lease infinite
!
interface GigabitEthernet0/0
 vrf forwarding Mgmt-vrf
 ip address 192.168.5.230 255.255.255.0
 negotiation auto
!
interface GigabitEthernet4/0/1
 no switchport
 ip address 10.0.0.2 255.255.255.0
!
interface GigabitEthernet4/0/2
 no switchport
 ip address 10.0.2.2 255.255.255.0
!
interface GigabitEthernet4/0/13
 switchport access vlan 6
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet4/0/14
 switchport access vlan 6
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet4/0/15
 switchport access vlan 6
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet4/0/16
 switchport access vlan 6
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/37
 switchport access vlan 2
 switchport trunk native vlan 2
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/38
 switchport access vlan 2
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/39
 switchport access vlan 3
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/40
 switchport access vlan 3
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/41
 switchport access vlan 4
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/42
 switchport access vlan 4
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/43
 switchport access vlan 5
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/44
 switchport access vlan 5
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/45
 switchport access vlan 6
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/46
 switchport access vlan 7
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/47
 switchport access vlan 8
 switchport mode access
 spanning-tree portfast
!
interface TenGigabitEthernet4/0/48
 switchport access vlan 8
 switchport mode access
 spanning-tree portfast
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan2
 ip address 192.168.1.1 255.255.255.0
!
interface Vlan3
 ip address 192.168.2.1 255.255.255.0
!
interface Vlan4
 ip address 192.168.3.1 255.255.255.0
!
interface Vlan5
 ip address 192.168.4.1 255.255.255.0
!
interface Vlan6
 ip address 192.168.5.1 255.255.255.0
!
interface Vlan7
 ip address 192.168.6.1 255.255.255.0
!
interface Vlan8
 ip address 192.168.7.1 255.255.255.0
 ip policy route-map SL
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
ip route 0.0.0.0 0.0.0.0 10.0.0.1
!
!
ip access-list extended SL
 10 deny   ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
 20 permit ip 192.168.7.0 0.0.0.255 any
!
!
route-map SL permit 10 
 match ip address access-list SL
 set ip next-hop 10.0.2.1
!

[Joseph W. Doherty]

Hmm, just wondering about your ACL, as it appears you're deny 192.168.0.0/16 before permitting 192.168.7.0/24.  As the latter is with the scope of the former, and all ACLs have an implicit deny all ACE, it appears your PBR wouldn't match anything.

Perhaps the ACL should only have its second ACE with a permit, or only that ACE made into deny with a following permit any any ACE.  (As prior thread is unknown to me and this thread doesn't fully describe topology, I cannot say which of the two revised ACLs should be used.)

Oh, BTW, I wouldn't recommend using infinite DHCP leases, as it sort of negates much of "dynamic".

[TheGoob]

Morning

From my own understanding, the Deny before the Permit was to block any of the other vlans from communicating with the vlan 8 [192.168.7.0/24] at it's Router level but stay at the Switch level, and then of course permit 192.168.7.0/24 so it could route to the Internet at that next-hope 10.0.2.1. I believe that intention was to keep the communication local. 

As far as the DHCP infinite, how funny! Every time I looked at that I said "well that is a weird command for dhcp considering it should be dynamic" but yeah never really looked into it, so I shall indeed remove that. 

BUT with that being said;

I mean I just have a vlan 192.168.1.0 which uses ISP 1 and a vlan2 192.168.2.0 which uses ISP2.. The Switch is 10G but when I xfer data from vlan 1 to vlan 2, it drops down to 1G, the Routers speed, and not across the Switch at 10G.. 

[Joseph W. Doherty]

Oops, my bad, the ACL's first ACE doesn't fully overlap the second ACE, because the first ACE has a /16 destination while the second ACE has an any destination.

However, rather than blocking traffic between other VLANs and VLAN 8, it's the first ACE that maintains normal routing.

[TheGoob]

I won't lie, I am unsure what you are telling me with that.. What I know is that what I have in place apparently isn't keeping the communication local to the Switch. So defeats the purpose of trying to utilize my 10G Capabilities. In my mind without really understanding how to implement, it seems I need to have the vlans "route" or communicate using the Switch IP as the GW but then how would they still refer to their ISPs. Hmm. 

I wonder if my PBR telling 192.168.7.0/24 to have next-hop as 10.0.2.1 [It's ISP Router] is causing it to route outside of the Switch and then back. 

[Joseph W. Doherty]

Okay, to be sure you understand, when you use an ACL with PBR, a permit match invokes the PBR clause it's used within.  A deny match mean the ACL didn't match, and the PBR clause continues onto the next PBR clause, if any.

Also keep in mind, basically, PBR is conditional static routing.

A PBR policy applies to interface ingress traffic.

Let's consider the following cases:

Traffic going to VLAN 8 that's 1) 192.168.0.0/16 => 192.168.7.0/24 and 2) not 192.168.0.0/16 => 192.168.7.0/24

and traffic from VLAN 8 that's 3) 192.168.7.0/24 => 192.168.0.0/16 and 4) 192.168.7.0/24 => not 192.168.0.0/16

Case 1:

First SL ACE matches but as it's a deny, normal routing should take place, i.e. packet routed to 192.168.7.0/24

Case 2:

Neither SL ACE matches, so it's an implicit deny, normal routing should take place, i.e. packet routed to 192.168.7.0/24

Case 3:

First SL ACE matches but as it's a deny, normal routing should take place, i.e. packet routed to 192.168.0.0/16

Case 4:

Second SL ACE matches but as it's a permit, packet routed to 10.0.2.1

BTW, as you're running the switch as L3, I'm also thinking whether

ip default-gateway 10.0.0.1
should be
ip route 0.0.0.0 0.0.0.0 10.0.0.1

Also BTW, looking at your other thread

ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/0

is often bad for Internet, you should use the next hop's IP.

[TheGoob]

Where are you seeing ‘default gateway 10.0.0.1’? On my most recent running-config shown it has the ‘ip route 0.0.0.0 0.0.0.0 10.0.0.1’. Gah I wonder if I am going mad. 
Also the ip route gigabit ethernet 0/0/0 was on the ISR, but can still change that . Also assuming next hop is the wan ip address itself. 

[Joseph W. Doherty]

Where are you seeing ‘default gateway 10.0.0.1’? 

Bottom of OP.

Also assuming next hop is the wan ip address itself. 

Yes, the other device that connects to your router.

[TheGoob]

I apologize, I had posted an updated running config about six or seven posts ago and had thought that I had the correct IP route.

[Joseph W. Doherty]

I apologize, I had posted an updated running config about six or seven posts ago and had thought that I had the correct IP route.

Ah, it's I who should apologize, as I do see that (now) in one of your following replies.

I wonder if Packet Tracer will allow me to reproduce the routing, because it's not obvious, to me, why it's not working correctly.

[TheGoob]

Again I still wonder if my explicit PBR making it’s (192.168.7.0) next-hop be its Router of 10.0.2.1 causes it to bypass the Switch and hit the ISR the back to the Switch.