Nexus 9K - hardware ip glean throttle

[melchib]

Hello-

 Just curious if anyone has used "hardware ip glean throttle" command on the Nexus 9K platform.  In our situation, we have VXLAN leaf switches with the anycast gateway configured and on some of them we are getting many, many L3 glean drops because of CoPP.  This is typically due to a end-system failing or going offline and other devices still trying to send traffic to it:

#show hardware rate-limiter 

Units for Config: kilo bits per second
Allowed, Dropped & Total: aggregated bytes since last clear counters


Module: 1
  R-L Class             Config              Allowed              Dropped                Total
 +----------------+----------+--------------------+--------------------+--------------------+
  L3 glean                 100               430174             31028576             31458750
 <snipped>

We have been advised by TAC to enable/configure the "hardware ip glean throttle" command and was mainly just curious if there are any drawbacks or caveats when enabling it.

Thanks.

Brad

[Andrea Testino]

You should not need this in a Nexus 9000 as the CoPP policy itself handles this.

Previously on our Nexus 7000s this was a heavily recommended feature as glean packets were handled in the default-class which is of a "match-any" type and is not very aggressive.

On Nexus 3000/9000s, there's a built-in class-map for Glean packets and it will be rate-limited accordingly depending on whether lenient/strict/dense/etc is being used.  

N77-A-Admin# show policy-map interface control-plane | beg glean

Versus:

N93180YC-EX# show policy-map interface control-plane | beg glean
      match exception glean
      set cos 1
      police cir 800 kbps , bc 32000 bytes 
      module 1 :
        transmitted 0 bytes;
        dropped 0 bytes;

Hope this helps.

[melchib]

Thanks for the reply Andrea.

The default hardware rate-limiter looks like its much more aggressive than CoPP for glean packets.  If I'm understanding it correctly, the hardware rate-limiter will start dropping after 100 kbps of glean traffic, where the default CoPP strict policy will only kick in after 800 kbps.  

Many of our switches we are seeing drops in both places - I should point out this environment is running vxlan bgp evpn with the distributed anycast gateway...below is an output from a leaf that has around 600 active IPs/VMs behind it.

SwitchA# show hardware rate-limiter 

Units for Config: kilo bits per second
Allowed, Dropped & Total: aggregated bytes since last clear counters


Module: 1
  R-L Class             Config              Allowed              Dropped                Total
 +----------------+----------+--------------------+--------------------+--------------------+
  L3 glean                 100              1391048           4015250130           4016641178

SwitchA# show policy-map interface control-plane
Control Plane

  Service-policy  input: copp-system-p-policy-strict

    class-map copp-system-p-class-l3uc-data (match-any)
      match exception glean
      set cos 1
      police cir 800 kbps , bc 32000 bytes 
      module 1 :
        transmitted 2110377225 bytes;
        dropped 1360768741 bytes;

Both hardware rate-limiter and CoPP showing drops...so question is where to tune?  The hardware ip glean throttle command does make sense if there is a end-system that goes offline/fails and other IP's try to continually send traffic.

Is there any drawback to enabling hardware ip glean throttle - I couldn't really find any in the searching I did.

Thanks again for the reply.

Brad

[Andrea Testino]

Brad,

No drawbacks. It will simply install a /32 drop adjacency in HW to stop sub-sequent packets for the same next-hop from reaching the SUP as you already know.

Thanks!