Hardening NX-OS Layer 3

[wj343]

I have several Nexus 3000-series devices deployed as Layer 3 switches for customer machines. Each switch has 40+ SVI's acting as the default gateway for a particular host:

interface Vlan110
  no shutdown
  ip address 10.0.0.1/24

interface Ethernet1/10
  switchport access vlan 110

It appears that the host address on each one of these SVI's can act as a management IP to the switch. For example, running a nmap on one IP shows a lot of available ports:

PORT      STATE    SERVICE     VERSION
21/tcp    filtered ftp
111/tcp   filtered rpcbind
161/tcp   open     snmp?
179/tcp   open     tcpwrapped
512/tcp   filtered exec
513/tcp   filtered login
514/tcp   filtered shell
2049/tcp  filtered nfs
8083/tcp  open     us-srv?
27000/tcp filtered flexlm0
32768/tcp filtered filenet-tms

Is there a (easy) way to lock down these IP's so only the mgmt0 port on the device has certain services enabled? I've considered using ACL's, but the dynamic nature of the environment makes it difficult to keep updated (these SVI's can change very often).

[f00z]

Yes, documentation is here: (control plane policing/copp)

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/security/503_u2_2/b_Cisco_n3k_security_cg_503_u2_2/b_Cisco_n3k_security_cg_503_u2_2_chapter_01101.html

[Sergiu.Daniluk]

HI @wj343 

There are two options:

1. As @f00z mentioned, you can change the CoPP (which is the control plane policing) configuration, by allowing 0 pps for SSH/Telnet class. Why this works? Because all inband is being forwarded through CoPP when packets are destined to CPU, while the mgmt0 (oob) is not policed.

Note: when modifying the CoPP, you will need to do it in a maintanance window as there might impact the control plane of the Nexus switch.

2. Option which I propose: configure an access list to permit traffic from any IP address only destined to the mgmt0 IP address, and apply this ACL to VTY lines:

Config on Nexus:
ip access-list permit_oob_only
  10 permit ip any 172.16.1.1/24 

line vty
  exec-timeout 600
  access-class permit_oob_only in

Result:
mgmt-sw# ssh admin@10.10.10.53
ssh: connect to host 10.10.10.53 port 22: Connection refused

172.16.1.1 = mgmt0 / oob IP address

10.10.10.53 = SVI

Hope it helps,

Sergiu

[wj343]

Thanks for your reply! I'm looking to secure more than just SSH (also NX-API, SNMP, NTP, ... etc.), so it looks like I will need to use the CoPP policer with pps 0 to accomplish this.

I did some initial testing with the CoPP policer, and it seems that it ends up dropping the packets completely without any response (ie. conn refused for TCP or ICMP unreachable for UDP). This ends up showing up on nmap:

Host is up, received arp-response (0.042s latency).
Not shown: 131037 closed ports
Reason: 65521 conn-refused and 65516 port-unreaches
PORT      STATE         SERVICE        REASON
21/tcp    filtered      ftp            no-response
111/tcp   filtered      rpcbind        no-response
161/tcp   filtered      snmp           no-response
512/tcp   filtered      exec           no-response
513/tcp   filtered      login          no-response
514/tcp   filtered      shell          no-response
2049/tcp  filtered      nfs            no-response
27000/tcp filtered      flexlm0        no-response
32765/tcp filtered      unknown        no-response
32767/tcp filtered      filenet-powsrm no-response
32768/tcp filtered      filenet-tms    no-response
44138/tcp filtered      unknown        no-response
56000/tcp filtered      unknown        no-response
56001/tcp filtered      unknown        no-response
21/udp    open|filtered ftp            no-response
67/udp    open|filtered dhcps          no-response
111/udp   open|filtered rpcbind        no-response
123/udp   open          ntp            udp-response
161/udp   open          snmp           udp-response
512/udp   open|filtered biff           no-response
513/udp   open|filtered who            no-response
514/udp   open|filtered syslog         no-response
547/udp   open|filtered dhcpv6-server  no-response
2049/udp  open|filtered nfs            no-response
3784/udp  open|filtered bfd-control    no-response
3785/udp  open|filtered unknown        no-response
27000/udp open|filtered unknown        no-response
32765/udp open|filtered unknown        no-response
32767/udp open|filtered unknown        no-response
32768/udp open|filtered omad           no-response
56000/udp open|filtered unknown        no-response
56001/udp open|filtered unknown        no-response
60176/udp open|filtered unknown        no-response

Is there any way you can think of to "mask" these ports? Obviously I want to reveal as little as possible about the system for security purposes.

[f00z]

Make your own control plane (Copp) ACLs. Make your own policy map and apply them inside of that plus some of the default copp rules. This is probably the hardest part of using a new platform, each nexus platform has a different set of rules and limits for copp and testing all this is a huge pain but once you have it set it's not something that changes often.

I set a lot of the unused portions of the default copp to 0, and then make my own acls for the management (what to allow for ssh, snmp, bgp, arp, nd, icmp , ospf etc).  Also disable any features you aren't using.

No-response is what you want, meaning it's completely hidden, doesn't even send back a tcp reset. With the proper copp setup it will be no response for everything unless you allow it. Good idea to allow icmp echo at least and of course arp/nd.

[wj343]

Why would no response be a better solution than tcp reset / icmp unreachable? When the majority of ports will return a tcp reset / icmp unreachable, those specific policed ports with no response will stand out like a sore thumb (see output from nmap above)

I did some more digging and found access to the bash shell for my Nexus 3k, and found that it has iptables included. However, it doesn't seem that any rules I create work even though they are applied in the correct vrf (default). Any chance I can do the policing setup from bash?

Default iptables rules:

bash-4.2# iptables -vnL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 local-hooks  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  veobc  *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  shreth *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  lo     *       0.0.0.0/0            0.0.0.0/0            multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768
    0     0 ACCEPT     tcp  --  veobc  *       0.0.0.0/0            0.0.0.0/0            multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768
    0     0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768
    0     0 ACCEPT     udp  --  lo     *       0.0.0.0/0            0.0.0.0/0            multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768
    0     0 ACCEPT     udp  --  veobc  *       0.0.0.0/0            0.0.0.0/0            multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768
    0     0 DROP       tcp  --  *      *      !127.0.0.1           !127.0.0.1            tcp dpt:44138

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain local-hooks (1 references)
 pkts bytes target     prot opt in     out     source               destination         
bash-4.2# 

[Sergiu.Daniluk]

Most of the services (nxapi, ntp, snmp etc)  can be simply disabled, either by "no feature <service>" - if this is a feature, or through the cmd (example: no ntp enable).

I would recommend you to look on the config guide. Also, this page might help: https://tools.cisco.com/security/center/resources/securing_nx_os.html 

Regards,

Sergiu

[wj343]

Hi,

I need to use these services, otherwise I would have disabled them already.

I've already gone through the configuration guide, but it doesn't look like it was targeted towards a layer 3 deployment like mine is. For example, I cannot get infrastructure ACL's to work properly on SVI's: I apply them to the SVI, but traffic originating from a subnet within the VLAN bypass the ACL completely.

[f00z]

Can't do it with iptables, that's an internal thing to the VM running the OS 

Have to use control plane policing, and define your own ACLs instead of using the ones supplied.  Half of them or so you can re-use because they are L2 stuff, but define most of your own for layer3.

[wj343]

Any idea how to use CoPP to restrict access to certain IP(s)?

For example, if I want to limit access to 179/tcp (bgp) to only certain IP's, I cannot just use a "police pps 0" like I do with other services that I intend to only access through the mgmt0 port.

[f00z]

If you have one in the lab you can play with , which is what I usually do. 

Remove everything from the policy map you don't need and then set this:

class-map copp-s-selfIp

in the policy map class to 0 pps.  And then make your own class maps and acls for what you want to allow. 

A good page to reference and see what some of these things are:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/security/6x/b_Cisco_n3k_Security_Config_6x/b_Cisco_n3k_Security_Config_6x_chapter_01011.html

[Sergiu.Daniluk]

So basically you need to use the services, but you want to restrict access to them only to a specifc subnet? Or you want to completely stop inband access to the services? Or just restrict access to the services from the security point of view?

In the end each service has it's own way of hardening the security. For example:

+ NXAPI can be restricted only to management vrf using "nxapi use-vrf vrf" command. This way only access through mgmt0 can work.

+ ssh/telnet can be restricted using access lists

+ SNMP/NTP - can be restricted through authentication.

What I am trying to say is that you have other mechanism of NXOS hardening compared with "masking" the services using SVIs.

Ofc, if this is some sort of requirements in your company to block inband access, then SVI ACLs, with self IP as dst IP in the entires, an specific allowed subnet as source, it should work.  I just test in in the lab. If for you does not work, please share the cfg on your SVI + the ACL and source IP, destination IP and protocol for your tests.

Regards,

Sergiu

[wj343]

SVI ACL's were originally not working for me, but it looks like that was a feature limitation on the switch.

I applied the following command and it looks like everything is working now!

ip access-list match-local-traffic 

I was hoping to have some other elegant solution rather than defining ACL's for every single IP on SVI's (I have dozens of constantly changing IP's), but I suppose this will do.

[f00z]

The whole point of COPP is so you don't have to put acls on each SVI that pertain to management traffic.  The SVI or port acls are for infrastructure protection, and COPP is to protect the individual device's control plane.  

When I said making ACLs, it's to allow certain IPs to the control plane, such as which IPs you want to connect with  BGP, SSH, etc.  This applies to everything on the device automatically (applies to all frames/packets which get punted to the cpu).

[wj343]

Alright ... I'm at a complete loss now as to what's happening.

I tried doing a test with router ACL's as follows:

ip access-list match-local-traffic

ip access-list ipv4-system-ingress
  statistics per-entry
  10 deny ip any xxx.xxx.xxx.249/32 
  20 permit ip any any 

interface Vlan30
  no shutdown
  ip access-group ipv4-system-ingress in
  ip address xxx.xxx.xxx.249/29
  ip verify unicast source reachable-via rx

interface Ethernet1/30
  switchport access vlan 30
  spanning-tree port type edge

Theoretically, this should prevent any host connected on Vlan 30 from sending traffic to xxx.xxx.xxx.249, no? I verified this by attempting to ping xxx.xxx.xxx.249 from a host within the Vlan, and it seemed to work. ACL statistics show packets being dropped as expected (this was after 6 ICMP requests):

# show ip access-lists ipv4-system-ingress

IP access list ipv4-system-ingress
        statistics per-entry 
        10 deny ip any xxx.xxx.xxx.249/32 [match=6] 
        20 permit ip any any [match=2359] 

However, after running a nmap to xxx.xxx.xxx.249 without changing any settings, it still ends up getting a response from 179/tcp (bgp), as well as showing other filtered ports:

Not shown: 976 closed ports
Reason: 976 resets
PORT      STATE    SERVICE     REASON      VERSION
21/tcp    filtered ftp         no-response
22/tcp    filtered ssh         no-response
111/tcp   filtered rpcbind     no-response
161/tcp   filtered snmp        no-response
179/tcp   open     tcpwrapped  syn-ack
512/tcp   filtered exec        no-response
513/tcp   filtered login       no-response
514/tcp   filtered shell       no-response
2049/tcp  filtered nfs         no-response
8083/tcp  filtered us-srv      no-response
27000/tcp filtered flexlm0     no-response
32768/tcp filtered filenet-tms no-response
58080/tcp filtered unknown     no-response
60020/tcp filtered unknown     no-response
60443/tcp filtered unknown     no-response
61532/tcp filtered unknown     no-response
61900/tcp filtered unknown     no-response
62078/tcp filtered iphone-sync no-response
63331/tcp filtered unknown     no-response
64623/tcp filtered unknown     no-response
64680/tcp filtered unknown     no-response
65000/tcp filtered unknown     no-response
65129/tcp filtered unknown     no-response
65389/tcp filtered unknown     no-response

It seems like some services are completely bypassing the ACL I setup, which is quite frustrating.