How to protect 6500 from this ? CoPP, hardware rate limiters, or others

gnijs · ‎05-04-2016

Hello,

We have had a broadcast storm inside a VLAN that impacted the CPU of a CAT6500 SUP720 switch.

The packet was an IPv4 UDP packet with broadcast MAC address FF:FF:FF:FF:FF:FF. The destination address was the local VLAN broadcast address: 10.10.10.255 (/24 subnet).

CPU went to 100%. The question now is how to pretect from this kind of packets ?

NOTE: broadcast storm control doesn't work, because we can't get the % low enough on 10GE links (even 0.01% is still 100 Mbps)
There were many packets (20.000/s) but they were very small (< 50 Mbps)

I would prefer to use the hardware rate limiters before the software CoPP policing, but which category does this packet fall under ?

         MCAST NON RPF   Off                  -       -     -
        MCAST DFLT ADJ   On              100000     100  Not sharing
      MCAST DIRECT CON   Off                  -       -     -
        ACL BRIDGED IN   Off                  -       -     -
       ACL BRIDGED OUT   Off                  -       -     -
           IP FEATURES   Off                  -       -     -
          ACL VACL LOG   On                2000       1  Not sharing
            MAC PBF IN   Off                  -       -     -
           CEF RECEIVE   Off                  -       -     -
             CEF GLEAN   Off                  -       -     -
      MCAST PARTIAL SC   On              100000     100  Not sharing
        IP RPF FAILURE   On                 100      10  Group:0 S
           TTL FAILURE   Off                  -       -     -
 ICMP UNREAC. NO-ROUTE   On                 100      10  Group:0 S
 ICMP UNREAC. ACL-DROP   On                 100      10  Group:0 S
         ICMP REDIRECT   Off                  -       -     -
           MTU FAILURE   Off                  -       -     -
       MCAST IP OPTION   Off                  -       -     -
       UCAST IP OPTION   Off                  -       -     -
           LAYER_2 PDU   Off                  -       -     -
            LAYER_2 PT   Off                  -       -     -
       LAYER_2 PORTSEC   Off                  -       -     -
     LAYER_2 MiniProto   Off                  -       -     -
      DHCP Snooping IN   Off                  -       -     -
     DHCP Snooping OUT   Off                  -       -     -
        ARP Inspection   Off                  -       -     -
             IP ERRORS   On                 100      10  Group:0 S
           CAPTURE PKT   Off                  -       -     -
            MCAST IGMP   Off                  -       -     -
 MCAST IPv6 DIRECT CON   Off                  -       -     -
 MCAST IPv6 ROUTE CNTL   Off                  -       -     -
 MCAST IPv6 *G M BRIDG   Off                  -       -     -
  MCAST IPv6 SG BRIDGE   Off                  -       -     -
  MCAST IPv6 DFLT DROP   Off                  -       -     -
 MCAST IPv6 SECOND. DR   Off                  -       -     -
  MCAST IPv6 *G BRIDGE   Off                  -       -     -
        MCAST IPv6 MLD   Off                  -       -     -
  IP ADMIS. ON L2 PORT   Off                  -       -     -
    UCAST IP TINY FRAG   Off                  -       -     -
    MCAST IP TINY FRAG   Off                  -       -     -
   UCAST UNKNOWN FLOOD   Off                  -       -     -
        MCAST IPv4 PIM   Off                  -       -     -

Packet properties:

interface Vl364, routine mistral_process_rx_packet_inlin, timestamp 14:40:39
dbus info: src_vlan 0x16C(364), src_indx 0x1000(4096), len 0x40(64)
  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x416C(16748)
  48820400 016C0000 10000100 40080000 00110448 02000008 00000008 416CBB18
mistral hdr: req_token 0x0(0), src_index 0x1000(4096), rx_offset 0x76(118)
  requeue 0, obl_pkt 0, vlan 0x16C(364)
destmac FF.FF.FF.FF.FF.FF, srcmac 00.50.56.94.79.51, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 41, identifier 26316
  df 0, mf 0, fo 0, ttl 128, src 10.10.10.113, dst 10.10.10.255
    udp src 47813, dst 47813 len 21 checksum 0xC22B

Carlos Villagran · ‎05-04-2016

Hi!

Can we try switch(config)#mls rate-limit unicast cef receive x x <<<< x is the number of pps you would like to allow and second x the max packets in burst allowed to be received.

Be aware of this command since ANY TRAFFIC DESTINED for the switch will be policied to the values configured. So, this is just a workaround meanwhile the actual root cause is discovered (who and why is 00.50.56.94.79.51 sending that traffic??)

Hope it helps, best regards!

JC

gnijs · ‎07-01-2016

Hello Carlos,

Thanks for you suggestion. I was hoping to use a more specific class instead of the "general" limit all packets to the CPU to a maximum. Anyway, we debugged the server and found the cause: the default gateway of the server was the broadcast address of the VLAN (a classic one)

regards,

Geert