Nexus 9K, Having trouble Implementing/Creating/Applying ACL's

[TheGoob]

Hello

I have a Nexus 9K w/6 vlans

GE 1/1-1/8

vlan1

vlan Interface 192.168.1.1 255.255.255.0

no shut

GE 1/9-1/16

vlan2

vlan Interface 192.168.2.1 255.255.255.0

no shut

GE 1/17-1/24

vlan3

vlan Interface 192.168.3.1 255.255.255.0

no shut

And so on for all 6.

I have ip route 0.0.0.0 0.0.0.0 192.168.1.2 [it's WAN Gateway] and so on to 192.168.3.2 [it's WAN Gateway] but have no issues with each vlan hitting the internet. My issues is I can not get any vlan to allow connectivity to another vlan, more specifically a host on one vlan can't communicate with a host on another. On the Nexus, I can Ping, from 1 vlan to another and back etc. So it is "there" but can not pass data. At this stage, I am fine with opening up ALL ACCESS ALLOW just to get my servers etc up and running.

My only familiarity with ACL's [manually] were my ASA-5508 and FPR1010, but it has been ages doing them CLI. I even tried using same formats as I had on those system but to no avail. I cant not find much reference on the subject, at least that I can understand/relate to.

I only just learned to 'feature interface vlan' so I am sure I am missing things as far as ACL's go. I looked into IP ACL and VLAN ACL, and nothing seems to click.

[MHM Cisco World]

So you add 

Feature inter vlan ?

If no add it 

Also share 

Show ip interface brief 

MHM

[TheGoob]

That’s what I am saying, in all of this I learned that I had to enable feature interface vlan, this is how I added the IP to that.

really I just wanna see working examples of ACL’s  because anything on google means nothing to what I’m trying to do apparently 

[TheGoob]

I was saying the vlans are set up and can each ping each others interface after the ‘feature interface vlan’ . Just need ACL guidance. So ACL’s need a feature option, so they need to be applied before working?

as far as the ip interface brief I’ll have to get you that when I am off of work for sure. 

[Ruben Cocheno]

@TheGoob 

Just check here for guidance while setting up ACLs https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_7x_chapter_01001.html

Note:No license is required to use IP ACLs. Any feature not included in a license package is bundled with the nx-os image and is provided at no extra charge to you. For an explanation of the Cisco NX-OS licensing scheme, see the Cisco NX-OS Licensing Guide.

[TheGoob]

Hello.

So last night when looking up ACL's and came across VACLS, IP ACLS I did indeed see that link. I sat for many moments trying to incorporate my needs with the examples.

I asked myself how do I make, for example;

192.168.3.15 [vlan3 w/ Interface vlan3 ip 192.168.3.1] get allowed access to 192.168.4.33 [vlan4 w/ Interface vlan4 ip 192.168.4.1] on Ports 22, 4443, 9090 with a name 'proxmox_acl_in'. 

Nothing could I conjure to figure that out based on the link.

So I then figured well I am a noob, theses guys earned their knowledge, so let's make it simple. Any ip on any vlan can access any ip on any vlan..for now.

I couldn't even make that work.

[Christopher_Starcher]

Hello, where are you applying the ACL and how are you applying it?  It looks like you have an SVI for each VLAN.

In general, the ACL is applied to the layer 3 interface on the router that owns the subnet.

Using your 9k post, here's an example:

int Vlan1

ip address 192.168.1.1 255.255.255.0

ip access-group aclnamewhatever in

ip access-list aclnamewhatever

permit ip 192.168.1.0/24 any

deny ip any any

This should effectively permit any inbound traffic from your Vlan1 SVI to go anywhere else.  But, in general, I would expect this to get you what you're looking for.  Then any device plugged into an interface on VLAN 1 with an address in that subnet should be allowed to go anywhere.

[TheGoob]

So I did re-create this, but in similar fashion to my system... I did have to do 'permit ip 192.168.3.0/24 any'.

I did so, but still no connection. I am curious though... On the interface vlan, I do ip access-group 'name' in, then have to exit out to (config) for access-list 'name' but how does access-list know to validate access-group and how does access-group know how to allow access-list? The only vlan related acc-* is the access-group, but unsure how it knows what -list to look up.

[TheGoob]

In truth I could not tell you at this point exactly what I tried, but it was a nasty little arrangement of variables that ultimately failed. Though, seeing what you wrote I can not recall doing that though.From what you are writing, it appears the ACL's are attached to a vlan you want to "leave" from as opposed to a vlan you want access to? So the above example would allow 192.168.2.0 192.168.6.0 IN access to 192.168.1.0?

[Christopher_Starcher]

Yeah, it's a little different to wrap your head around.  But, ACLs applied to SVI's are handled differently than those applied to physical interfaces.  If you've configured an SVI like my example, the ACL will permit traffic sourced from your VLAN 1 subnet, which is 192.168.0.0/24, destined to any of your other VLANs and the open internet.

That's why I asked what interface your ACL was applied to and how you applied it.  But, I'm assuming your VLAN interface is on the 9k and 192.168.1.1 is the default gateway for all of your VLAN 1 devices.  That 9K has the route going to your next-hop, 192.168.1.2/32.  But, I am making a few assumptions here...  Am I understanding things correctly?  I'm happy to help by providing some examples for you.  But, my examples won't work if you're trunking VLAN 1 to your next-hop and that router is where VLAN 1's default gateway lives.  I hope all this is making sense...

If I understand correctly, then that ACL will work if applied to an SVI that's on the 9k.  GIve it a shot when you get a chance and let's see what happens..

[TheGoob]

We have it possibly not the norm, but what we ended u with was this.

The Nexus has 6 vlans, 1-6, and each vlan Interface has an IP address in relation to its vlan [vlan1, 192.168.1.1, vlan2 192.168.2.1]. I have my FPR [WAN Facing Router] with 6 vlans [matching the Nexus vlan], assigned to 6 Interfaces, and their IP is 192.168.1.2, 192.168.2.2 and so on. My FPR is also handing out DHCP IP's. We have this set up so that all vlans could communicate at Nexus Port Speeds of 10Gbps.

I have ip routes for each Network to the Internet, 0.0.0.0 0.0.0.0 192.168.1.2, 0.0.0.0 0.0.0.0 192.168.2.2 and so on. This way all VLAN Traffic stays on Nexus to utilize 10Gbps and then when they need the Internet they hit the FPR through the x.x.x.2 IP's. I guess for simplicity sake, I was just focusing on the Nexus  vlan to vlan ACL's.  All vlan Interfaces are are x.x..1.1, x.x.2.1, x.x.3.1 I did for SVI for Routing.

No trunking or port channel here.

[TheGoob]

So in creating this, here is what I did. Mind you, right now, my intentions with this was to assume it will give Anything on vlan4, 192.168.3.0 any and all access to 'any' other vlan also on Nexus [which all are]

So, my 192.168.3.115 Windows should be able to SSH into 192.168.1.180. In this test, I made only vlan2 (192.168.1.0) and vlan 4 (192.168.3.0) configured on the Nexus..

Only Access-list

ip access-list ceyea_to_all
  10 permit ip 192.168.3.0/24 any
  20 deny ip any any

IP Routes so they can hit the Internet

ip route 0.0.0.0/0 192.168.1.2
ip route 0.0.0.0/0 192.168.2.2
ip route 0.0.0.0/0 192.168.3.2

Vlan 2 and 4 configuration with IP's [ONLY vlan 4 has the access-group]

interface Vlan2
  no shutdown
  ip address 192.168.1.1/24

interface Vlan4
  no shutdown
  ip access-group ceyea_to_all in
  ip address 192.168.3.1/24

That is all and only ACL/Access-group I have... Assuming that was all I need to allow 192.168.3.0 touch and connect to anything, but in this case just 192.168.1.0. It did not.

[Christopher_Starcher]

Honestly, I'm thinking this isn't an ACL issue and it's more along the lines of layer 2 or 3.

On the 9k would you mind posting the output from these commands, please?

sh ip ro
sh ip int bri
sh ip accsess-list summary
sh mac address vlan 4
sh mac address vlan 2

You can hide the MAC addresses or just delete them from the output, I'm just interested in the interface info.

[TheGoob]

Morning

sh ip ro
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 1/0
    *via 192.168.3.2, [1/0], 11:41:06, static
192.168.1.0/24, ubest/mbest: 1/0, attached
    *via 192.168.1.1, Vlan2, [0/0], 00:00:59, direct
192.168.1.1/32, ubest/mbest: 1/0, attached
    *via 192.168.1.1, Vlan2, [0/0], 00:00:59, local
192.168.3.0/24, ubest/mbest: 1/0, attached
    *via 192.168.3.1, Vlan4, [0/0], 11:41:06, direct
192.168.3.1/32, ubest/mbest: 1/0, attached
    *via 192.168.3.1, Vlan4, [0/0], 11:41:06, local

 sh ip int bri

IP Interface Status for VRF "default"(1)
Interface            IP Address      Interface Status
Vlan2                192.168.1.1     protocol-up/link-up/admin-up
Vlan4                192.168.3.1     protocol-up/link-up/admin-up

sh ip access-lists summary

IPV4 ACL ceyea_to_all
        Total ACEs Configured: 2
        Configured on interfaces:
                Vlan4 - ingress (Router ACL)
        Active on interfaces:
                Vlan4 - ingress (Router ACL)
IPV4 ACL copp-system-p-acl-auto-rp
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-bgp
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-dhcp
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-dhcp-relay-response
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-eigrp
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-ftp
        Total ACEs Configured: 4
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-hsrp
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-http
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-https
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-icmp
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-igmp
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-msdp
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-ntp
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-openflow
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-ospf
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-pim
        Total ACEs Configured: 3
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-pim-mdt-join
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-pim-reg
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-ptp
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-ptp-uc
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-radius
        Total ACEs Configured: 8
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-rip
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-sftp
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-snmp
        Total ACEs Configured: 4
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-ssh
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-tacacs
        Total ACEs Configured: 2
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-telnet
        Total ACEs Configured: 4
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-tftp
        Total ACEs Configured: 4
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-traceroute
        Total ACEs Configured: 3
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-undesirable
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-vpc
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL copp-system-p-acl-vrrp
        Total ACEs Configured: 1
        Configured on interfaces:
        Active on interfaces:
IPV4 ACL sl_def_acl
        Statistics enabled
        Total ACEs Configured: 4
        Configured on interfaces:
        Active on interfaces:

 sh mac address vlan 4
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*    4     1.2.3.4   dynamic  0         F      F    Eth1/33
G    4     5.6.7.9  static       -          F      F    sup-eth1(R)

 sh mac address vlan 2
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*    2     1.2.4.5   dynamic  0         F      F    Eth1/18
*    2     5.6.7.8   dynamic  0         F      F    Eth1/17
G    2     9.7.5.4   static   -         F      F    sup-eth1(R

show interface brief

--------------------------------------------------------------------------------
Port   VRF          Status IP Address                              Speed    MTU
--------------------------------------------------------------------------------
mgmt0  --           down   172.21.9.7                              --       1500

--------------------------------------------------------------------------------
Ethernet        VLAN    Type Mode   Status  Reason                 Speed     Por
t
Interface                                                                    Ch
#
--------------------------------------------------------------------------------
Eth1/1          1       eth  access up      none                     1000(D) --
Eth1/2          1       eth  access down    Link not connected       auto(D) --
Eth1/3          1       eth  access up      none                      10G(D) --
Eth1/4          1       eth  access down    Link not connected       auto(D) --
Eth1/5          1       eth  access up      none                     1000(D) --
Eth1/6          1       eth  access down    Link not connected       auto(D) --
Eth1/7          1       eth  access down    Link not connected       auto(D) --
Eth1/8          1       eth  access down    Link not connected       auto(D) --
Eth1/9          1       eth  access up      none                     1000(D) --
Eth1/10         1       eth  access down    Link not connected       auto(D) --
Eth1/11         1       eth  access down    Link not connected       auto(D) --
Eth1/12         1       eth  access down    Link not connected       auto(D) --
Eth1/13         1       eth  access down    Link not connected       auto(D) --
Eth1/14         1       eth  access down    Link not connected       auto(D) --
Eth1/15         1       eth  access down    Link not connected       auto(D) --
Eth1/16         1       eth  access down    Link not connected       auto(D) --
Eth1/17         2       eth  access up      none                     1000(D) --
Eth1/18         2       eth  access up      none                     1000(D) --
Eth1/19         2       eth  access down    Link not connected       auto(D) --
Eth1/20         2       eth  access down    Link not connected       auto(D) --
Eth1/21         2       eth  access down    Link not connected       auto(D) --
Eth1/22         2       eth  access down    Link not connected       auto(D) --
Eth1/23         2       eth  access down    Link not connected       auto(D) --
Eth1/24         2       eth  access down    Link not connected       auto(D) --
Eth1/25         3       eth  access up      none                     1000(D) --
Eth1/26         3       eth  access up      none                      10G(D) --
Eth1/27         3       eth  access down    Link not connected       auto(D) --
Eth1/28         3       eth  access down    Link not connected       auto(D) --
Eth1/29         3       eth  access down    Link not connected       auto(D) --
Eth1/30         3       eth  access down    Link not connected       auto(D) --
Eth1/31         3       eth  access down    Link not connected       auto(D) --
Eth1/32         3       eth  access down    Link not connected       auto(D) --
Eth1/33         4       eth  access up      none                     1000(D) --
Eth1/34         4       eth  access down    Link not connected       auto(D) --
Eth1/35         4       eth  access down    Link not connected       auto(D) --
Eth1/36         4       eth  access down    Link not connected       auto(D) --
Eth1/37         4       eth  access down    Link not connected       auto(D) --
Eth1/38         4       eth  access down    Link not connected       auto(D) --
Eth1/39         4       eth  access down    Link not connected       auto(D) --
Eth1/40         4       eth  access down    Link not connected       auto(D) --
Eth1/41         5       eth  access up      none                     1000(D) --
Eth1/42         5       eth  access up      none                     1000(D) --
Eth1/43         5       eth  access down    Link not connected       auto(D) --
Eth1/44         5       eth  access down    Link not connected       auto(D) --
Eth1/45         5       eth  access down    Link not connected       auto(D) --
Eth1/46         5       eth  access down    Link not connected       auto(D) --
Eth1/47         5       eth  access down    Link not connected       auto(D) --
Eth1/48         5       eth  access down    Link not connected       auto(D) --
Eth1/49         6       eth  access up      none                     1000(D) --
Eth1/50         6       eth  access up      none                      10G(D) --
Eth1/51         6       eth  access down    Link not connected       auto(D) --
Eth1/52         6       eth  access down    Link not connected       auto(D) --
Eth1/53         6       eth  access down    Link not connected       auto(D) --
Eth1/54         6       eth  access down    Link not connected       auto(D) --
Eth1/55         6       eth  access down    Link not connected       auto(D) --
Eth1/56         6       eth  access down    Link not connected       auto(D) --

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan1     --                                      down   Administratively down
Vlan2     --                                      up     --
Vlan4     --                                      up     --

[Christopher_Starcher]

Thank you, it looks like you have a few interfaces in the VLANs you're working on.  The ACL looks to be applied correctly and the SVI's look to be right.  I don't see a L2/3 problem.  The traffic you're testing between the two networks won't use any of the default routes, it's local to local and should be routed between the two VLANs without leaving the 9k.

Based on the output, I'm not seeing the reason why a device in VLAN 4 isn't able to SSH to a device in VLAN 2 using that ACL.

At this point, I'd enable logging on the ACL entries, both the permit and the deny all, then test again.  But, from what I'm seeing, this should be working.

Just to verify, SSH between the Windows hosts works without an ACL in place, correct?