05-12-2020 03:47 PM
I have several Nexus 3000-series devices deployed as Layer 3 switches for customer machines. Each switch has 40+ SVI's acting as the default gateway for a particular host:
interface Vlan110 no shutdown ip address 10.0.0.1/24 interface Ethernet1/10 switchport access vlan 110
It appears that the host address on each one of these SVI's can act as a management IP to the switch. For example, running a nmap on one IP shows a lot of available ports:
PORT STATE SERVICE VERSION 21/tcp filtered ftp 111/tcp filtered rpcbind 161/tcp open snmp? 179/tcp open tcpwrapped 512/tcp filtered exec 513/tcp filtered login 514/tcp filtered shell 2049/tcp filtered nfs 8083/tcp open us-srv? 27000/tcp filtered flexlm0 32768/tcp filtered filenet-tms
Is there a (easy) way to lock down these IP's so only the mgmt0 port on the device has certain services enabled? I've considered using ACL's, but the dynamic nature of the environment makes it difficult to keep updated (these SVI's can change very often).
05-12-2020 04:18 PM
Yes, documentation is here: (control plane policing/copp)
05-12-2020 11:37 PM
HI @wj343
There are two options:
1. As @f00z mentioned, you can change the CoPP (which is the control plane policing) configuration, by allowing 0 pps for SSH/Telnet class. Why this works? Because all inband is being forwarded through CoPP when packets are destined to CPU, while the mgmt0 (oob) is not policed.
Note: when modifying the CoPP, you will need to do it in a maintanance window as there might impact the control plane of the Nexus switch.
2. Option which I propose: configure an access list to permit traffic from any IP address only destined to the mgmt0 IP address, and apply this ACL to VTY lines:
Config on Nexus:
ip access-list permit_oob_only 10 permit ip any 172.16.1.1/24 line vty exec-timeout 600 access-class permit_oob_only in Result: mgmt-sw# ssh admin@10.10.10.53 ssh: connect to host 10.10.10.53 port 22: Connection refused
172.16.1.1 = mgmt0 / oob IP address
10.10.10.53 = SVI
Hope it helps,
Sergiu
05-15-2020 08:45 AM
Thanks for your reply! I'm looking to secure more than just SSH (also NX-API, SNMP, NTP, ... etc.), so it looks like I will need to use the CoPP policer with pps 0 to accomplish this.
I did some initial testing with the CoPP policer, and it seems that it ends up dropping the packets completely without any response (ie. conn refused for TCP or ICMP unreachable for UDP). This ends up showing up on nmap:
Host is up, received arp-response (0.042s latency). Not shown: 131037 closed ports Reason: 65521 conn-refused and 65516 port-unreaches PORT STATE SERVICE REASON 21/tcp filtered ftp no-response 111/tcp filtered rpcbind no-response 161/tcp filtered snmp no-response 512/tcp filtered exec no-response 513/tcp filtered login no-response 514/tcp filtered shell no-response 2049/tcp filtered nfs no-response 27000/tcp filtered flexlm0 no-response 32765/tcp filtered unknown no-response 32767/tcp filtered filenet-powsrm no-response 32768/tcp filtered filenet-tms no-response 44138/tcp filtered unknown no-response 56000/tcp filtered unknown no-response 56001/tcp filtered unknown no-response 21/udp open|filtered ftp no-response 67/udp open|filtered dhcps no-response 111/udp open|filtered rpcbind no-response 123/udp open ntp udp-response 161/udp open snmp udp-response 512/udp open|filtered biff no-response 513/udp open|filtered who no-response 514/udp open|filtered syslog no-response 547/udp open|filtered dhcpv6-server no-response 2049/udp open|filtered nfs no-response 3784/udp open|filtered bfd-control no-response 3785/udp open|filtered unknown no-response 27000/udp open|filtered unknown no-response 32765/udp open|filtered unknown no-response 32767/udp open|filtered unknown no-response 32768/udp open|filtered omad no-response 56000/udp open|filtered unknown no-response 56001/udp open|filtered unknown no-response 60176/udp open|filtered unknown no-response
Is there any way you can think of to "mask" these ports? Obviously I want to reveal as little as possible about the system for security purposes.
05-15-2020 10:35 AM
Make your own control plane (Copp) ACLs. Make your own policy map and apply them inside of that plus some of the default copp rules. This is probably the hardest part of using a new platform, each nexus platform has a different set of rules and limits for copp and testing all this is a huge pain but once you have it set it's not something that changes often.
I set a lot of the unused portions of the default copp to 0, and then make my own acls for the management (what to allow for ssh, snmp, bgp, arp, nd, icmp , ospf etc). Also disable any features you aren't using.
No-response is what you want, meaning it's completely hidden, doesn't even send back a tcp reset. With the proper copp setup it will be no response for everything unless you allow it. Good idea to allow icmp echo at least and of course arp/nd.
05-15-2020 01:57 PM
Why would no response be a better solution than tcp reset / icmp unreachable? When the majority of ports will return a tcp reset / icmp unreachable, those specific policed ports with no response will stand out like a sore thumb (see output from nmap above)
I did some more digging and found access to the bash shell for my Nexus 3k, and found that it has iptables included. However, it doesn't seem that any rules I create work even though they are applied in the correct vrf (default). Any chance I can do the policing setup from bash?
Default iptables rules:
bash-4.2# iptables -vnL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 1 60 local-hooks all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- veobc * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- shreth * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT tcp -- lo * 0.0.0.0/0 0.0.0.0/0 multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768 0 0 ACCEPT tcp -- veobc * 0.0.0.0/0 0.0.0.0/0 multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768 0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768 0 0 ACCEPT udp -- lo * 0.0.0.0/0 0.0.0.0/0 multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768 0 0 ACCEPT udp -- veobc * 0.0.0.0/0 0.0.0.0/0 multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768 0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 56000,56001,21,111,512,513,514,2049,27000,2049,32765,32767,32768 0 0 DROP tcp -- * * !127.0.0.1 !127.0.0.1 tcp dpt:44138 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain local-hooks (1 references) pkts bytes target prot opt in out source destination bash-4.2#
05-15-2020 03:16 PM
Most of the services (nxapi, ntp, snmp etc) can be simply disabled, either by "no feature <service>" - if this is a feature, or through the cmd (example: no ntp enable).
I would recommend you to look on the config guide. Also, this page might help: https://tools.cisco.com/security/center/resources/securing_nx_os.html
Regards,
Sergiu
05-15-2020 03:26 PM
Hi,
I need to use these services, otherwise I would have disabled them already.
I've already gone through the configuration guide, but it doesn't look like it was targeted towards a layer 3 deployment like mine is. For example, I cannot get infrastructure ACL's to work properly on SVI's: I apply them to the SVI, but traffic originating from a subnet within the VLAN bypass the ACL completely.
05-15-2020 05:37 PM
Can't do it with iptables, that's an internal thing to the VM running the OS
Have to use control plane policing, and define your own ACLs instead of using the ones supplied. Half of them or so you can re-use because they are L2 stuff, but define most of your own for layer3.
05-15-2020 06:46 PM
Any idea how to use CoPP to restrict access to certain IP(s)?
For example, if I want to limit access to 179/tcp (bgp) to only certain IP's, I cannot just use a "police pps 0" like I do with other services that I intend to only access through the mgmt0 port.
05-15-2020 09:49 PM
If you have one in the lab you can play with , which is what I usually do.
Remove everything from the policy map you don't need and then set this:
class-map copp-s-selfIp
in the policy map class to 0 pps. And then make your own class maps and acls for what you want to allow.
A good page to reference and see what some of these things are:
05-16-2020 12:52 AM
So basically you need to use the services, but you want to restrict access to them only to a specifc subnet? Or you want to completely stop inband access to the services? Or just restrict access to the services from the security point of view?
In the end each service has it's own way of hardening the security. For example:
+ NXAPI can be restricted only to management vrf using "nxapi use-vrf vrf" command. This way only access through mgmt0 can work.
+ ssh/telnet can be restricted using access lists
+ SNMP/NTP - can be restricted through authentication.
What I am trying to say is that you have other mechanism of NXOS hardening compared with "masking" the services using SVIs.
Ofc, if this is some sort of requirements in your company to block inband access, then SVI ACLs, with self IP as dst IP in the entires, an specific allowed subnet as source, it should work. I just test in in the lab. If for you does not work, please share the cfg on your SVI + the ACL and source IP, destination IP and protocol for your tests.
Regards,
Sergiu
05-16-2020 03:40 AM
SVI ACL's were originally not working for me, but it looks like that was a feature limitation on the switch.
I applied the following command and it looks like everything is working now!
ip access-list match-local-traffic
I was hoping to have some other elegant solution rather than defining ACL's for every single IP on SVI's (I have dozens of constantly changing IP's), but I suppose this will do.
05-16-2020 12:23 PM
The whole point of COPP is so you don't have to put acls on each SVI that pertain to management traffic. The SVI or port acls are for infrastructure protection, and COPP is to protect the individual device's control plane.
When I said making ACLs, it's to allow certain IPs to the control plane, such as which IPs you want to connect with BGP, SSH, etc. This applies to everything on the device automatically (applies to all frames/packets which get punted to the cpu).
05-16-2020 12:39 PM
Alright ... I'm at a complete loss now as to what's happening.
I tried doing a test with router ACL's as follows:
ip access-list match-local-traffic ip access-list ipv4-system-ingress statistics per-entry 10 deny ip any xxx.xxx.xxx.249/32 20 permit ip any any interface Vlan30 no shutdown ip access-group ipv4-system-ingress in ip address xxx.xxx.xxx.249/29 ip verify unicast source reachable-via rx interface Ethernet1/30 switchport access vlan 30 spanning-tree port type edge
Theoretically, this should prevent any host connected on Vlan 30 from sending traffic to xxx.xxx.xxx.249, no? I verified this by attempting to ping xxx.xxx.xxx.249 from a host within the Vlan, and it seemed to work. ACL statistics show packets being dropped as expected (this was after 6 ICMP requests):
# show ip access-lists ipv4-system-ingress IP access list ipv4-system-ingress statistics per-entry 10 deny ip any xxx.xxx.xxx.249/32 [match=6] 20 permit ip any any [match=2359]
However, after running a nmap to xxx.xxx.xxx.249 without changing any settings, it still ends up getting a response from 179/tcp (bgp), as well as showing other filtered ports:
Not shown: 976 closed ports Reason: 976 resets PORT STATE SERVICE REASON VERSION 21/tcp filtered ftp no-response 22/tcp filtered ssh no-response 111/tcp filtered rpcbind no-response 161/tcp filtered snmp no-response 179/tcp open tcpwrapped syn-ack 512/tcp filtered exec no-response 513/tcp filtered login no-response 514/tcp filtered shell no-response 2049/tcp filtered nfs no-response 8083/tcp filtered us-srv no-response 27000/tcp filtered flexlm0 no-response 32768/tcp filtered filenet-tms no-response 58080/tcp filtered unknown no-response 60020/tcp filtered unknown no-response 60443/tcp filtered unknown no-response 61532/tcp filtered unknown no-response 61900/tcp filtered unknown no-response 62078/tcp filtered iphone-sync no-response 63331/tcp filtered unknown no-response 64623/tcp filtered unknown no-response 64680/tcp filtered unknown no-response 65000/tcp filtered unknown no-response 65129/tcp filtered unknown no-response 65389/tcp filtered unknown no-response
It seems like some services are completely bypassing the ACL I setup, which is quite frustrating.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide