10-25-2019 01:20 PM
I have cisco 6500 WS-SUP720-3BXL that I’m testing control-plane policing and it seems to be working, I see packet drops, but the CPU is still overwhelmed to 90%. I’m using an online stresser, basically a DDOS attack test. When I test with an LDAP amplification attack to the Cisco 6500 with control plane protection the CPU gets overwhelmed, no matter what I do. The stresser will send about 1 gig of traffic for 30 seconds to the 6500 CPU at which point it becomes unresponsive and disconnects my cli connection. We use this as a LAN switch and have a much power ASR at the edge.
I was able to get around the issue by rate limiting LDAP packets to the 6500 on the ASR link that is connected to the 6500, however It doesn’t seem like control-plane policing on the 6500 is effective at stopping attacks. Even if I drop all LDAP packets to the CPU with CPP.
Class CP_ldap_class
police cir 32000 bc 1500
conform-action drop
exceed-action drop
Is there something I’m doing wrong here or is this just older technology, hardware limitation, that can’t handle that much traffic to the cpu even with CPP? I’m not even sure how the CPP dropping all traffic differs from just rate limiting on the CPU, not sure if there is a difference and how its handled. Clearly dropping all traffic to the CPU with CPP is not working and neither is plain RL. I tried looking for other alternatives like hardware punt rate limits, but you can’t configure hardware RL for LDAP.
TIA, Paul
10-25-2019 01:50 PM
Hello, hmm it looks like there's something not working in the policy if you cannot even full drop traffic.
did you try also with police rate pps rather than cir bps ?
something like:
Router(config-pmap-c)# police rate 5 pps conform-action drop exceed-action drop
Are you sure that packets are matched ?
Is the policy applied inbound under control-plane ? show policy-map control-plane
Regards
10-28-2019 06:28 AM
Hello Paul,
You might want to review the restrictions here:
"CoPP is not be enabled in hardware unless you have enabled PFC QoS globally with the mls qos command. If you do not enter the mls qos command, CoPP is not hardware-accelerated."
Do you have mls qos enabled?
Hope that helps!
10-28-2019 09:36 AM
Hi, brselzer, mls qos is enabled on the 6500, below is my CP policy-map
Policy Map CPP_policy
Description: Block DOS to the Control-plane
Class CP_ldap_class
police cir 32000 bc 1500
conform-action drop
exceed-action drop
Class CP_udp_class
police cir 200000 bc 6250
conform-action transmit
exceed-action drop
Class CP_icmp_class
police cir 500000 bc 15625
conform-action transmit
exceed-action drop
Class CP_arp_class
police cir 500000 bc 15625
conform-action transmit
exceed-action drop
Class CP_udp_local_class
Class class-default
GW#show class-map CP_ldap_class
Class Map match-all CP_ldap_class (id 16)
Match access-group name CP_ldap_access
GW#sh ip access-lists CP_ldap_access
Extended IP access list CP_ldap_access
20 permit udp any eq 389 any (1172423 matches)
When I test with a simulated LDAP DDOS attack the RP CPU spikes to 90% kicking me out of the CLI. I know the policy map is working because when i flood the CP with icmp it drops the excess packets, however when i send about 1 Gig of DDOS LDAP packets over to the CP, the CPU goes over 90%. I Do see the policy-map rate limit packets or drop them outright as currently configured by the policy map, conform-action transmit exceed-action drop. However the 6500 RP CPU still spikes. Is this a case of too much traffic, about 1 gig, to the CPU and not able to drop packets fast enough?
Paul
10-28-2019 08:29 AM
10-28-2019 10:04 AM
Hi Joseph,
I dont currently have access to newer code for the 6500. I'm using the 6500 as an access switch right now. When looking at this issue i can see the CP policy-map drop packets sources from the LDAP port, see class map below. The issue is the 6500 RP goes to 90% even while the policy-map, ACL show does packets as dropped. RP cpu goes to 90% and I lose the CLI. This could be a bug as Im using an older version of IOS, s72033-advipservicesk9_wan-mz.122-33.SXJ7.bin.
The ICMP RL on the CP works, I can see icmp packets being dropped and timing out so I know the polcy-map is configured correctly but when i send over 1 gig of LDAP DDOS traffic even when the policy dropping all those packets the CPU goes way up. So it could be a problem with the IOS but I'm not sure.
I'm assuming when you have CoPP that the hardware drops the packets even before they hit the RP/software but obviously this doesn't seem to be the case, at least not with a simulated DDOS using LDAP or DNS based UDP amplifications attacks. Should the 6500 be able to drop that much traffic before it hits the RP.
Paul
Policy Map CPP_policy
Description: Block DOS to the Control-plane
Class CP_ldap_class
police cir 32000 bc 1500
conform-action drop
exceed-action drop
Class CP_udp_class
police cir 200000 bc 6250
conform-action transmit
exceed-action drop
Class CP_icmp_class
police cir 500000 bc 15625
conform-action transmit
exceed-action drop
Class CP_arp_class
police cir 500000 bc 15625
conform-action transmit
exceed-action drop
Class CP_udp_local_class
Class class-default
GW#show class-map CP_ldap_class
Class Map match-all CP_ldap_class (id 16)
Match access-group name CP_ldap_access
GW#sh ip access-lists CP_ldap_access
Extended IP access list CP_ldap_access
20 permit udp any eq 389 any (1172423 matches)
10-28-2019 10:28 AM
Hello Paul,
Config looks right. I would expect CoPP to be able to drop any volume of traffic without any issues if it was working correctly. You would start to overrun the interfaces at some point but hardware CoPP should still be working if it was in hardware. Seems like for some reason the box can't handle the logic in hardware and has to punt to CPU to assist. As others have mentioned, might be worth trying to upgrade if you can get your hands on it as that version of software went EOL earlier this year. Lots of fixes since that version was released.
Hope that helps!
07-14-2020 09:05 AM
Late reply to this issue, but it may help someone else. When you use online stresser website to generate LDAP Amplification attack, the resulting traffic stream to target IP address is usually a mix of UDP source port 389 traffic AND UDP fragments, which have no port numbers just IP addresses. If you are policing the UDP source port 389 traffic only, then you are still getting hit with the UDP fragments.
Try adding the line “permit udp any any fragments” before your “permit udp any eq 389 any” line and see if that makes a difference.
07-14-2020 10:01 AM
I have since upgraded the hardware but what you suggestion makes sense to me. I do believe that is what happened and your solution could be the right one.
thanks for your reply.
Paul
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: