ACL "workaround" for Nexus not supporting 'default next-hop' command.

[TheGoob]

Hello.

I have a working PBR configuration that I currently have no issues with except for the fact that it seems to bypass/overlook 'inter-vlan' connectivity and won't route locally, which I really need. My version of NXOS does not support default next-hop [i've looked] but I have seen some examples of making it work using ACL's? The examples I have seen I have mentally been unable to insert into my situation. Anyone familiar with how to keep PBR active but also allow inter-vlan routing to work as intended as well? Here is my current PBR;

ip access-list vlan2ip
  10 permit ip 192.168.1.0 0.0.0.255 any
ip access-list vlan3ip
  10 permit ip 192.168.2.0 0.0.0.255 any
ip access-list vlan4ip
  10 permit ip 192.168.3.0 0.0.0.255 any
ip access-list vlan5ip
  10 permit ip 192.168.4.0 0.0.0.255 any
ip access-list vlan6ip
  10 permit ip 192.168.6.0 0.0.0.255 any
ip access-list vlan7ip
  10 permit ip 192.168.5.0 0.0.0.255 any

vlan 1-7

route-map vlan2map permit 10
  match ip address vlan2ip
  set ip next-hop 192.168.1.2
route-map vlan3map permit 10
  match ip address vlan3ip
  set ip next-hop 192.168.2.2
route-map vlan4map permit 10
  match ip address vlan4ip
  set ip next-hop 10.0.0.1
route-map vlan5map permit 10
  match ip address vlan5ip
  set ip next-hop 10.0.0.1
route-map vlan6map permit 10
  match ip address vlan6ip
  set ip next-hop 10.0.0.1
route-map vlan7map permit 10
  match ip address vlan7ip
  set ip next-hop 10.0.0.1

interface Vlan2
  no shutdown
  ip address 192.168.1.1/24
  ip policy route-map vlan2map
  ip dhcp relay address 192.168.1.2

interface Vlan3
  no shutdown
  ip address 192.168.2.1/24
  ip policy route-map vlan3map
  ip dhcp relay address 192.168.2.2

interface Vlan4
  no shutdown
  ip address 192.168.3.1/24
  ip policy route-map vlan4map
  ip dhcp relay address 10.0.0.1

interface Vlan5
  no shutdown
  ip address 192.168.4.1/24
  ip policy route-map vlan5map
  ip dhcp relay address 10.0.0.1

interface Vlan6
  no shutdown
  ip address 192.168.6.1/24
  ip policy route-map vlan6map
  ip dhcp relay address 10.0.0.1

interface Vlan7
  no shutdown
  ip address 192.168.5.1/24
  ip policy route-map vlan7map
  ip dhcp relay address 10.0.0.1

[liviu.gheorghe]

Hello @TheGoob ,

can you elaborate on why you need the PBR with the set ip default option? I cannot seem to get the idea from the description and the configuration.

[TheGoob]

Hello

Well, unless I have implemented PBR wrong, the purpose was for vlan 2-3 to route back to their source, which is really through a Trunk from FPR to Nexus, and then vlan 4-7 to route back to their source, on the ISR via 10.0.0.1 IP. 
Any style of 0.0.0.0 0.0.0.0 .destination. Did not work, so I went PBR.  Problem is with implementing PBR, the nexus doesn’t route locally first, so there is no inter-vlan.

To be honest I just wanted to have vlan 2-3 TRUNK back to the FPR (which it is) but also vlan 4-7 TRUNK back to the ISR, but no matter what I did they would not route back correctly.

I even made each network a default route back. This too is why I went PBR

[balaji.bandi]

Hope one of your post  i have mentioned below comment :

As i have mentioned in the main post in the forum - you need to add in ACL deny for the other VLAN IP address to traffic locally between VLAN (that is my main view to leave the static IP routing to towards ISR (bit testing required here for the correct outcome)

this means since you have next hop all the source, it send all packets to next hop to make decision, to over come that you need to deny in the ALC for Intra-vlan traffic to flow locally

[liviu.gheorghe]

Can you share an updated topology of your network? I recall seeing it in one of your posts drawn by balanji.bandi I think. I searched for it but didn't find it.

[TheGoob]

This is the best I have at this location.

Essentially, vlan 2-3 are being "sourced" from FPR and vlan 4-7 are being "sourced" from ISR. On NEXUS, ALL vlans MUST route locally on Nexus to maintain 10GBps speed but I also need/require vlan 2-3 to route "out" to FPR because of email in/out and also NGINX Proxy etc and I need vlan 4-7 to route "out" to ISR.

My only option, that I could find, was to PBR 2-3 and PBR 4-7 so they have their own "out. UNFORTUNATELY this seems to also negate inter-vlan [local] routing.

I did try it doing this; Hving vlan 2-3 TRUNK back to FPR and 4-7 TRUNK back to ISR, which would not require any PBR And would route locally via SVI's, BUT I could not get the default route out to work, nothing did, may latest was;

192.168.1.0 - 192.168.2.0 255.255.255.0 192.168.1.2

192.168.3.0 - 192.168.6.0 255.255.255.0 192.168.5.2

Everything just sat there and did nothing to get out to the Internet.

Anyway, I kinda of like the PBR and all works fine, but not it doesn't route locally and have to set up ACL's on FPR and ISR just to access a local database, not cool.

[Giuseppe Larosa]

Hello @TheGoob ,

in order to be able to perform inter VLAN routing on the Nexus and to use PBR for internet access you need to change each ACL in the following manner:

ip access-list vlan2ip
  10 deny 192.168.1.0 0.0.0.255 192.168.0.0 0.0.255.255
  20 permit ip 192.168.1.0 0.0.0.255 any

The deny statement placed before the current statement will not drop traffic but it will make inter VLAN traffic to be not processed by PBR and to be destination based routed that is what you want.

Hope to help

Giuseppe

[TheGoob]

Hello

10 deny ip 192.168.1.0 0.0.0.255 192.168.0.0 0.0.255.255

Deny is not supported on PBR. Please check your configuration.

[liviu.gheorghe]

There is no problem configuring the access list and PBR like @Giuseppe Larosa suggested:

nx9000-1# sho access-list vlan2ip

IP access list vlan2ip
10 deny ip 192.168.1.0 0.0.255.255 192.168.0.0 0.0.255.255
20 permit ip 192.168.1.0 0.0.0.255 any
nx9000-1# sho route-map
route-map test, permit, sequence 10
Match clauses:
ip address (access-lists): vlan2ip
Set clauses:
ip next-hop 182.168.1.1
nx9000-1#

[TheGoob]

Well that was my next question TBH. My "current" vlan2ip access-list is designated to the PBR.. I assume by me trying to modify it it is denying my ability to do so, because it is part of PBR? Are you suggesting I make a new ACL?

This is my current

ip access-list vlan2ip   
10 permit ip 192.168.1.0 0.0.0.255 any 

route-map vlan2map permit 10   
match ip address vlan2ip   
set ip next-hop 192.168.1.2

I get the error why editing current ACL.

[Giuseppe Larosa]

Hello @TheGoob ,

The NXOS CLI parser checks that the ACL is in use, you have to either remove the PBR application under the SVI or to create a new ACL with a different name

Hope to help

Giuseppe

[TheGoob]

Morning.

So last night I did a test run and for the vlan2ip (the SVI interface, the acl the route-map) I deleted everything and then created route-map, interface SVI and the acl with deny then permit. All fine, but when I apply the ACL or should I said route-map?) to the svi it again says “deny not supported on pbr” you guys are saying it is so forgive my (again) silly ignorance but, how?

You then mention create a new acl. That is easy enough…. But how does it get applied to the SVI along with the existing acl that is already applied? Will the nexus routing see both ACL’s? The one applied ON the interface and the ACL just sitting on the nexus saying deny?

[Giuseppe Larosa · Accepted solution]

Hello @TheGoob ,

try the following

we create two ACLs with permit statement

ip access-list vlan2-to-others   
10 permit ip 192.168.1.0 0.0.0.255 192.168.0.0 0.0.255.255 

ip access-list vlan2ip   
10 permit ip 192.168.1.0 0.0.0.255 any 

 We rewrite the route-map to use two clauses the first block with no set action it will use destination based routing

route-map vlan2map permit 5   
match ip address vlan2-to-others   

route-map vlan2map permit 10   
match ip address vlan2ip   
set ip next-hop 192.168.1.2

  Try this way there may be some HW limitations for the Nexus

Hope to help

Giuseppe

[TheGoob]

Well that did it. All vlans on Nexus route to one another on Nexus utilizing 10G speeds [well, darn close] and I verified as such because I deleted the ACL on the FPR which was how I had to do it prior to.

SO thank you.

I apologize for the pages upon pages, but when it doesn't click, I can't give up, I have to keep asking and researching, or else I would quit.