Yes it all works the way we expect it to now.
Yea!
I would never have imagine the default addition to the route-map was the key difference on how it routes.
Well, in the original PBR policy, an attempt was made to distinguish between allowing VLAN 8 to route between it and the other local VLANs/subnets normally, and only direct its non local network (Internet) traffic to the new link.
Part of the key to understanding, the replacement PBR policy, its "default next-hop" basically applies to just overriding the default route. This is exactly what we wanted. We didn't want all VLAN 8 traffic directed to the new external path.
In truth I did not purposely select PBR but was not sure how to route in my scenario . . .
Yea, that's normally a problem, because you selectively want some specific traffic (based on source and destination) to flow along one path, and other some specific traffic (also based on source and destination) to flow across another path (Actually, if you had your own public Internet IP, and two ISPs, you could direct your outbound traffic to an ISP of your choice [much as we're doing now] but the return traffic might flow back on either ISP without your preference of which is used. (Since you're not using your own public IP, i.e. it's ISP specific, traffic will flow back across the same ISP it went outbound upon.)
Also another reason I did not want to do load balancing or failover scenario was ISP2 is my wifes and ISP1 was mine, I did not want anything I do to slow hers down...
Laugh! I understand, high level, executive policy decisions, which can, from time to time, preempt specific technical (better?, optimal?, robust?) approaches. ; )
I hope you were fine with spending so much time trying to recreate the scenario, that is crazy it was hours long, I am thankful to you for doing so.
Rest assured, I am fine with the time I spent. It was very much a learning experience for me. Laugh, one of my niche services, during my career, was working on solving issues that no one else could. Didn't always succeed, but succeeded enough which is why I would be assigned such cases.
In this case, one of the reasons it took me so long, because, at first glance, what you were trying to do looked okay to me. Further, the other replies you received earlier, thought the same. I.e. @Enes Simnica and @elwin-berrar thought what you had was fine.
As how the PBR policy should work, was sort of locked into my mine, i.e. the destination IP, in an ACL's ACE, should be considered, I was looking for some other reason why the PBR policy wasn't working. So, I ran lots of various tests trying to identify where the failure was. Heck, I even try testing with number ACLs rather than your named ACLs, because way back when, I recalled when named ACLs were introduced, some features that used ACLs, wouldn't process them. I also wondered, whether the original PBR policy would work as we all thought it should (i.e. even if destination IPs not ignored).
I.e. given:
ip access-list extended SL
10 deny ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
20 permit ip 192.168.7.0 0.0.0.255 any
route-map SL permit 10
match ip address access-list SL
set ip next-hop 10.0.2.1
Does the ACE 10 cause the PBR match to not match, or does that fact the ACE does match, i.e. you didn't hit the bottom of the ACL, count as a PBR "match"?
So I also tested something like:
ip access-list extended SL1
10 permit ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
ip access-list extended SL2
20 permit ip 192.168.7.0 0.0.0.255 any
route-map SL deny 10
match ip address access-list SL1
route-map SL permit 20
match ip address access-list SL2
set ip next-hop 10.0.2.1
The above didn't work either, again (as I later figured out) because the destination IP was being ignored.
With the above, I also enable policy debugging, but it's output was somewhat unclear, but seemed to show, ACL SL1 was matching undesired traffic (basically, same result as if it changed ACE 10 to "permit ip192.168.0.0 0.0.255.255 any".
With that, and more focused Internet questions, I finally saw mention of ACE destination IPs being ignored when using PBR.
Also, in my "research" of using PBR, I didn't find a matching example case like yours.
Anyway, with the understanding of the destination IP being ignored, I then had to figure out if there was a way to do it using PBR. I found the "default next-hop" option, with it cryptic explanation, for what it does, and tested it. Eureka!
Also anyway, part of my time also was taken in considering if what you described was a bug. Even though I was able to obtain the same result in CML (all my devices were routers, i.e. no switches, no PC hosts), I wondered how like CML might be like Packet Tracer, which often works incorrectly doing "complex" things. Yet, as I did get the same results, I figured, hmm, maybe we're just missing something "basic".
So, all the above took some time. But, again, a good learning experience for me.
