cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

BGP Flowspec implementation on NCS5500 platforms

4636
Views
5
Helpful
2
Comments
Cisco Employee

Feature Description

 

 


BGP flowspec in a nutshell is a feature that will allow you to receive IPv4/IPv6 traffic flow specification (source X, destination Y, protocol UDP, source port A .. etc) and actions that need to be taken on that traffic (drop, or police .. or redirect etc) via BGP update.
Inside the BGP update, the flowspec matching criteria is represented by BGP NLRI and the actions are represented by BGP extended communities.


This feature is based on RFC 5575 and can be used to mitigate against DDoS attack. When a certain host inside of a network is being attacked, we can send a flowspec update to edge routers so that attack traffic can be policed or dropped, or even redirected elsewhere, maybe to an appliance that can clean the traffic (filter out the bad traffic and forward only the good traffic toward the affected host).

Once flowspecs have been received by a router and programmed in applicable line cards, any active L3 ports on those line cards will start processing ingress traffic according to flowspec rules.
If needed, we can disable flowspec processing on specific ports of the LC via CLI config (discussed later).
Note also that flowspec will only affect ingress traffic, it won't interfere with egress direction.

Flowspec can be programmed on different kind of interfaces:

  1. regular interface, e.g. TenGigE0/0/0/0
  2. vlan interface, e.g. TenGigE0/0/0/0.1 , Bundle-Ether3.2
  3. bundle interface, e.g. Bundle-Ether3

Ingress traffic can be matched by many criteria as defined in the RFC.
We can define only one, some, or all criteria to match traffic.
Once a set of criteria in a flowspec is defined, then all criteria must match the packet for actions to take place.

Available criteria:

     

How about a case where we receive multiple different flowspecs from controller matching the same traffic stream?
Say the controller sends us flowspec type2 (source address), type3 (protocol), and type11 (DSCP) each in different update; and later we send a single traffic stream matching all three flowspec rules.
Which flowspec will match the traffic in hardware?
Answer:
The traffic will match the flowspec with lowest numeric type value.
In this case, traffic will match flowspec type 2 (because 2<3<11).
As a result, whatever action being defined along with flowspec type 2 will be taken.

Here's an example.
Say we receive following flowspecs from controller.

RP/0/RP0/CPU0:NCS5500#sh flowspec ipv4 detail         
Fri Jun 15 12:58:01.387 CEST
 
AFI: IPv4
  Flow           :Source:14.84.0.1/32
    Actions      :Traffic-rate: 7000000 bps  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :                   0/0                  
      Transmitted         :                   0/0                  
      Dropped             :                   0/0                  
  Flow           :Proto:=255
    Actions      :Traffic-rate: 5000000 bps  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :                   0/0                  
      Transmitted         :                   0/0                  
      Dropped             :                   0/0                  
  Flow           :DSCP:=20
    Actions      :Traffic-rate: 2000000 bps  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :                   0/0                  
      Transmitted         :                   0/0                  
      Dropped             :                   0/0     

Then we send traffic stream ingressing the router:
source 14.84.0.1
proto: 255
dscp: decimal 20 (af22)

This will result in the traffic matching flowspec type2 (source address):

RP/0/RP0/CPU0:NCS5500#sh flowspec ipv4 detail         
Fri Jun 15 13:05:13.312 CEST
 
AFI: IPv4
  Flow           :Source:14.84.0.1/32
    Actions      :Traffic-rate: 7000000 bps  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :                   8333868/2670601302                  
      Transmitted         :                   4612813/1402458454                 
      Dropped             :                   3721055/1268142848                 
  Flow           :Proto:=255
    Actions      :Traffic-rate: 5000000 bps  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :                   0/0                  
      Transmitted         :                   0/0                  
      Dropped             :                   0/0                  
  Flow           :DSCP:=20
    Actions      :Traffic-rate: 2000000 bps  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :                   0/0                  
      Transmitted         :                   0/0                  
      Dropped             :                   0/0     

We call the router that receives the BGP update as the "client", and the router/appliance that advertises the update as the "server" or "controller".
A router than runs IOS XR can function as either client or server, and even as both server/client at the same time. For instance, when the router also needs to take actions on the attack traffic in addition to propagating the flowspec rules to clients.

BGP flowspec feature has been supported on ASR9K and CRS for a while, and NCS5500 supports the feature starting with 6.5.1 release.
Disclaimer:
6.5.1 is not a GA release.

The following is comparison of flowspec support between existing XR routers like ASR9K and NCS5500.

Matching criteria:

fs_flowspec_type.png


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 



Actions:
 

fs_flowspec_action.png


This article will not discuss further details about working of BGP flowspec, because we already have great documentations that does that already:

1. basic description of BGP flowspec on ASR9K:
https://supportforums.cisco.com/t5/service-providers-documents/asr9000-xr-understanding-bgp-flowspec-bgp-fs/ta-p/3139916

2. detailed description of BGP flowspec on ASR9K, complete with config examples.

Please refer to :

Slides in PDF: https://buff.ly/2GE67rn

MP4 recording of the session: https://buff.ly/2J3gpzp



Rather, we will focus more on specific information pertaining to NCS5500 platform.

This is also a live document, we will update if new info becomes available.




Supported Hardware

    
If the router just needs to act as server only (no packet processing is required, perhaps the router is not in the attack path):
Any NCS5500 will do.

If the router needs to act as client (packets processing is required), we have two options:

  • NCS5500 modular platform
    Line card that RECEIVES traffic must have external TCAM and equipped with "Jericho +" or better ASIC.
    At the time of writing, only one LC satisfies above requirement: "NC55-36X100G-A-SE".
    Line card that TRANSMITS traffic can be of any flavor.
  • NCS5500 non modular platform
    Same constraint as above: the platform must have external TCAM and equipped with "Jericho +" or better ASIC.
    At the time of writing, only one NCS5500 non modular chassis satisfies above requirement: "NCS-55A1-36H-SE-S".

 

Also note that when the router acts as client, it doesn't matter either on which LC that you receive BGP update on, the LC that receives BGP update from BGP peer can be of any flavor.



Supported Scale

       
3,000 flowspecs (a "flowspec" consists of packet-matching criteria and desired actions)
Where a single flowspec can't take more than 1,024 TCAM entries on the LC.

A single flowspec can take more TCAM entries than others, for instance when we define some kind of IP address range instead of a host address as matching criteria.

 



Config example
"attack traffic ingressing on interface in Global Routing Table (GRT)"


Use case:

Attack IPv4 and IPv6 traffic received on GRT interface.
Redirect attack IPv4 traffic to VRF "honeypot" for scrubbing.
Redirect attack IPv6 traffic to different NH IPv6 address in GRT for scrubbing.

Note:

  • When redirecting traffic to different VRF (VRF "honeypot" in this use case), then that different VRF must have a route toward the traffic destination. Otherwise traffic will be redirected to that VRF internally but our router won't know where to forward it to.
    We can add this route via static route under VRF "honeypot", as shown in following config example.
  • When redirecting traffic to different NH IP address and we use XR router as controller, then that controller router must have a route towards the NH IP address. Otherwise the flowspec will not be advertised to clients.
    This requirement applies to both IPv4 and IPv6 flowspec.

 

Topology:

attack traffic
|
|
|
|
|
\/
GRT interface
NCS5500 ----------> redirect attack traffic to VRF or different NH IP address in GRT
|   |
|   |
|   +------ iBGP ----- flowspec server
|
|
\/
attacked host


Client config

... mandatory if we want to have flowspec IPv6 support (need LC reload after commit)
... this will configure support on all supported line cards on the chassis

hw-module profile flowspec v6-enable

... or this, if only want to configure support on specific LC

hw-module profile flowspec v6-enable location <>

 

... activate flowspec programming in the LC

flowspec
 local-install interface-all
!

... or if you want to be totally explicit:
    
flowspec
 address-family ipv4
  local-install interface-all
  !
  address-family ipv6
   local-install interface-all
   !
 !
        
... optional, disable flowspec processing on specific ingress interfaces

interface Bundle-Ether3.1
 ipv4 flowspec disable
 ipv6 flowspec disable
!
        
... basic PASS-ALL and DROP-ALL BGP policy

route-policy PASS-ALL
  pass
end-policy
!

route-policy DROP-ALL
  drop
end-policy
!

... configure BGP toward flowspec server
    
router bgp <>
 nsr
 bgp router-id <>
 address-family ipv4 flowspec
 !
 address-family ipv6 flowspec
 !
 neighbor <>
  remote-as <>
  address-family ipv4 flowspec
   route-policy PASS-ALL in
   route-policy DROP-ALL out
  !
  address-family ipv6 flowspec
   route-policy PASS-ALL in
   route-policy DROP-ALL out
  !
  update-source <>
 !
!

... define VRF to redirect the attack traffic to

vrf honeypot
 address-family ipv4 unicast
  import route-target
   4787:13
  !
  export route-target
   4787:13
  !
 !
 address-family ipv6 unicast
  import route-target
   4787:13
  !
  export route-target
   4787:13
  !
 !
!

... Define static route to forward the redirected traffic under VRF.
... Here we assume the traffic destination is any hosts under 70.0.0.0/8

router static
 vrf honeypot
  address-family ipv4 unicast
   70.0.0.0/8 200.255.55.2
  !
 !
!




Server Config

... basic PASS-ALL and DROP-ALL BGP policy

route-policy PASS-ALL
  pass
end-policy
!

route-policy DROP-ALL
  drop
end-policy
!

... configure BGP toward flowspec client

router bgp <>
 address-family ipv4 flowspec
 !
 address-family ipv6 flowspec
 !
 neighbor <>
  remote-as <>
  address-family ipv4 flowspec
   route-policy DROP-ALL in
   route-policy PASS-ALL out
  !
  address-family ipv6 flowspec
   route-policy DROP-ALL in
   route-policy PASS-ALL out
  !
 !
!

... let's configure flowspecs to be advertised to client

...... address-family ipv4 flowspec

    class-map type traffic match-all ipv4_fragment
     match destination-address ipv4 70.2.1.1 255.255.255.255
     match source-address ipv4 80.2.1.1 255.255.255.255
     match packet length 700
     match dscp af21
     match fragment-type  is-fragment
     end-class-map
    !
    
    class-map type traffic match-all ipv4_icmp
     match destination-address ipv4 70.2.1.1 255.255.255.255
     match source-address ipv4 80.2.1.1 255.255.255.255
     match packet length 700
     match dscp af21
     match fragment-type  is-fragment
     match ipv4 icmp-type 3
     match ipv4 icmp-code 2
     end-class-map
    !

    policy-map type pbr scale_ipv4
     class type traffic ipv4_fragment
      drop
     !
     class type traffic ipv4_icmp
      police rate 1 mbps
      !
      set dscp cs2
      redirect nexthop route-target 4787:13
     !
     class type traffic class-default
     !
     end-policy-map
    !

    flowspec
     address-family ipv4
      service-policy type pbr scale_ipv4
     !
    !

...... address-family ipv6 flowspec

    class-map type traffic match-all ipv6_tcp
     match destination-address ipv6 70:1:1::5a/128
     match source-address ipv6 80:1:1::5a/128
     match protocol tcp
     match destination-port 22
     match source-port 4000
     match tcp-flag 0x10
     match packet length 300
     match dscp af12
     end-class-map
    !

    class-map type traffic match-all ipv6_icmp
     match destination-address ipv6 70:2:1::1/128
     match source-address ipv6 80:2:1::1/128
     match packet length 800
     match dscp af22
     match ipv6 icmp-type 4
     match ipv6 icmp-code 1
     end-class-map
    !

    policy-map type pbr scale_ipv6
     class type traffic ipv6_tcp
      police rate 1 mbps
      !
      set dscp cs1
      redirect ipv6 nexthop 202:158:2::1
     !
     class type traffic ipv6_icmp
      police rate 1 mbps
      !
      set dscp cs3
      redirect nexthop route-target 4787:13
     !
     class type traffic class-default
     !
    !

    flowspec
     address-family ipv6
      service-policy type pbr scale_ipv6
     !
    !

 



Config example
"attack traffic ingressing on VRF interface"


Use case:


Attack traffic received on VRF "customer_1" interface.
Redirect this traffic to VRF "dirty-dancing" for scrubbing.

Note:

If we're redirecting traffic to different VRF (VRF "dirty-dancing" in this use case), then that different VRF must have a route toward the traffic destination. Otherwise traffic will be redirected to that VRF internally but our router won't know where to forward it to.
In this usecase, we will forward the traffic under VRF "dirty-dancing" using L3VPN route.
Our router gets this route from another L3VPN PE that is connected to redirect destination, so we won't need to define static routing under VRF like previous use case.



Topology:

attack traffic
|
|
|
|
|
\/
VRF "customer_1" interface
NCS5500 ----------> redirect attack traffic to different VRF "dirty-dancing"
|   |
|   |
|   +------ iBGP ----- flowspec server
|
|
\/
attacked host (VRF "customer_1")


Client Config

... mandatory if we want to have flowspec IPv6 support (need LC reload after commit)
... this will configure support on all supported line cards on the chassis

hw-module profile flowspec v6-enable

... or this, if only want to configure support on specific LC

hw-module profile flowspec v6-enable location <>


... activate flowspec programming in the LC

flowspec
 local-install interface-all
!
    
... or if you want to be totally explicit:
    
flowspec
 vrf customer_1
  address-family ipv4
   local-install interface-all
  !
  address-family ipv6
   local-install interface-all
  !
 !
!
        
... optional, disable flowspec processing on specific ingress interfaces

interface Bundle-Ether3.1
 ipv4 flowspec disable
 ipv6 flowspec disable
!
        
... basic PASS-ALL and DROP-ALL BGP policy

route-policy PASS-ALL
  pass
end-policy
!

route-policy DROP-ALL
  drop
end-policy
!

... configure BGP toward flowspec server
    
router bgp <>
 nsr
 bgp router-id <>
 address-family vpnv4 unicast
 !
 address-family vpnv6 unicast
 !
 address-family vpnv4 flowspec
 !
 address-family vpnv6 flowspec
 !
 neighbor <>
  remote-as <>
  address-family vpnv4 unicast
   route-policy PASS-ALL in
   route-policy PASS-ALL out
  !
  address-family vpnv6 unicast
   route-policy PASS-ALL in
   route-policy PASS-ALL out
  !
  address-family vpnv4 flowspec
   route-policy PASS-ALL in
   route-policy DROP-ALL out
  !
  address-family vpnv6 flowspec
   route-policy PASS-ALL in
   route-policy DROP-ALL out
  !
  update-source <>
 !
 vrf customer_1
  rd auto
  address-family ipv4 unicast
   label mode per-vrf
   redistribute connected
  !
  address-family ipv6 unicast
   label mode per-vrf
   redistribute connected
  !
  address-family ipv4 flowspec
  !
  address-family ipv6 flowspec
  !
 !
 vrf dirty_dancing
  rd auto
  address-family ipv4 unicast
   label mode per-vrf
  !
  address-family ipv6 unicast
   label mode per-vrf
  !
 !
!

... define VRF "customer_1" and "dirty_dancing"

vrf customer_1
 address-family ipv4 unicast
  import route-target
   4787:1313
  !
  export route-target
   4787:1313
  !
 !
 address-family ipv4 flowspec
  import route-target
   4787:1313
  !
  export route-target
   4787:1313
  !
 !
 address-family ipv6 unicast
  import route-target
   4787:1313
  !
  export route-target
   4787:1313
  !
 !
 address-family ipv6 flowspec
  import route-target
   4787:1313
  !       
  export route-target
   4787:1313
  !
 !
!
vrf dirty_dancing
 address-family ipv4 unicast
  import route-target
   4787:666
  !
  export route-target
   4787:666
  !
 !
 address-family ipv6 unicast
  import route-target
   4787:666
  !
  export route-target
   4787:666
  !
 !
!


Server Config

... basic PASS-ALL and DROP-ALL BGP policy

route-policy PASS-ALL
  pass
end-policy
!
route-policy DROP-ALL
  drop
end-policy
!

... configure BGP toward flowspec client

router bgp <>
 nsr
 bgp router-id <>
 address-family vpnv4 unicast
 !
 address-family vpnv6 unicast
 !
 address-family vpnv4 flowspec
 !
 address-family vpnv6 flowspec
 !
 neighbor <>
  remote-as <>
  address-family vpnv4 unicast
   route-policy PASS-ALL in
   route-policy PASS-ALL out
  !
  address-family vpnv6 unicast
   route-policy PASS-ALL in
   route-policy PASS-ALL out
  !
  address-family vpnv4 flowspec
   route-policy DROP-ALL in
   route-policy PASS-ALL out
  !
  address-family vpnv6 flowspec
   route-policy DROP-ALL in
   route-policy PASS-ALL out
  !
  update-source <>
 !
 vrf customer_1
  rd auto
  address-family ipv4 unicast
   label mode per-vrf
   redistribute connected
  !
  address-family ipv6 unicast
   label mode per-vrf
   redistribute connected
  !
  address-family ipv4 flowspec
  !
  address-family ipv6 flowspec
  !
 !
!

... let's configure flowspecs to be advertised to client

 

Note:

For a flowspec that is configured under VRF like the following, we can only use "redirect nexthop route-target <>" on the controller side to redirect this traffic to other VRF.
"redirect ipv4|ipv6 nexthop <>" is not supported in a VRF context.



...... address-family vpnv4 flowspec

    class-map type traffic match-all l3vpn_ipv4_attack
     match destination-address ipv4 202.158.3.2 255.255.255.255
     match source-address ipv4 202.158.1.2 255.255.255.255
     end-class-map
    !

    policy-map type pbr pm_cust_VRF_to_diff_VRF
     class type traffic l3vpn_ipv4_attack
      redirect nexthop route-target 4787:666
      set dscp cs6
      police rate 250 mbps
     !
     class type traffic class-default
     !
     end-policy-map
    !

    flowspec
     vrf customer_1
      address-family ipv4
       service-policy type pbr pm_cust_VRF_to_diff_VRF
     !
    !
    
...... address-family vpnv6 flowspec

    class-map type traffic match-all l3vpn_ipv6_attack
     match destination-address ipv6 200:158:3::2/128
     match source-address ipv6 200:158:1::2/128
     match protocol tcp
     match destination-port 22
     match source-port 4000
     match packet length 300
     match dscp af12
     end-class-map
    !

    policy-map type pbr pm_cust_VRF_to_diff_VRF_ipv6
     class type traffic l3vpn_ipv6_attack
      redirect nexthop route-target 4787:666
      set dscp cs6
      police rate 250 mbps
     !
     class type traffic class-default
     !
     end-policy-map
    !

    flowspec
     vrf customer_1
      address-family ipv6
       service-policy type pbr pm_cust_VRF_to_diff_VRF_ipv6
     !
    !


more detailed config examples to define packet matching criteria can be found here:

https://supportforums.cisco.com/t5/service-providers-blogs/ncs5500-bgp-flowspec-packet-matching-criteria/ba-p/3387457

 

 


BGP Flowspec and IPv6 BGP neighbors

 

The use-cases and config examples provided previously are when we have BGP neighbors that runs on IPv4.
What about if we have IPv6 BGP neighbors instead of IPv4?
We can still advertise flowspec to that neighbor, but only for address-family "ipv6 flowspec".

The following flowspec address-families are not supported:

  • ipv4 flowspec
  • vpnv4 flowspec
  • vpnv6 flowspec

This is by design and same limitation applies to other platforms that run IOS-XR like ASR9000.

 



BGP Flowspec and bundle interface

 

As mentioned before, BGP flowspec is also supported on bundle interface.

Note the following bundle behavior when it comes to flowspec traffic policing, however.

Say, we have interface Bundle-Ether3 with member links Hu0/4/0/0 and Hu0/4/0/9.
The member links are served by different NPU on LC 0/4/CPU0.

Hu0/4/0/0 will be served by NPU 0.
Hu0/4/0/9 will be served by NPU 1.

RP/0/RP0/CPU0:fretta-50#sh controllers npu voq-usage interface Hu0/4/0/0  instance all location 0/4/cpu0

-------------------------------------------------------------------
Node ID: 0/4/CPU0
Intf         Intf     NPU NPU  PP   Sys   VOQ   Flow   VOQ    Port
name         handle    #  core Port Port  base  base   port   speed
             (hex)                                     type        
----------------------------------------------------------------------
Hu0/4/0/0    2000130   0   1   21  2421   1024   4312 local   100G
RP/0/RP0/CPU0:fretta-50#

RP/0/RP0/CPU0:fretta-50#sh controllers npu voq-usage interface Hu0/4/0/9  instance all location 0/4/cpu0

-------------------------------------------------------------------
Node ID: 0/4/CPU0
Intf         Intf     NPU NPU  PP   Sys   VOQ   Flow   VOQ    Port
name         handle    #  core Port Port  base  base   port   speed
             (hex)                                     type        
----------------------------------------------------------------------
Hu0/4/0/9    2000178   1   1   21  2521   1096   4312 local   100G

And let's say there is currently a flowspec being advertised to our router:

policy-map type pbr scale_ipv4
 class type traffic ipv4_icmp
  police rate 1 mbps
  !
 !
 class type traffic class-default
 !
 end-policy-map
!

Now what's going to happen is that the police action will be programmed to both NPU 0 and NPU 1.
Hu0/4/0/0 on NPU 0 will police at 1Mbps.
Hu0/4/0/9 on NPU 1 will police at 1Mbps.

Assuming we receive many ICMP streams such that the streams will be load-balanced pretty evenly on both member links, this will give you 2Mbps police rate total instead of 1Mbps.

 



BGP Flowspec and BGP route-policy

We can configure BGP route-policy to filter flowspec updates.
One use case is maybe to limit flowspec updates with only specific subnet masks.

For instance:

router bgp <>
!
address-family ipv4 flowspec
!
address-family ipv6 flowspec
!
neighbor <>
  remote-as <>
  update-source <>
  address-family ipv4 flowspec
   route-policy routes_flowspec in
   route-policy drop-all out
  !
  ...

route-policy routes_flowspec
  if destination in ge_17 then
    pass
  endif
end-policy

prefix-set ge_17
  0.0.0.0/0 ge 17
end-set
!

Just be careful when doing route-policy.
If you commit the above config, and the coming flowspec updates don't have Type 1 (Destination address) criteria, then no updates would match the policy and it would be implicit deny for all flowspec updates.

 

 

BGP Flowspec and ACL

When flowspec is implemented on an interface that is also having ingress ACL, ACL will come before flowspec processing.

 

  • When the ACL is permitting the traffic:
    Flowspec will take whatever passed by ACL and run flowspec processing.
  • When the ACL is denying the traffic:
    Flowspec will not process any traffic since ACL has discarded them.

 

 

 BGP Flowspec and local QoS configuration

When flowspec is implemented on an interface that is also having local QoS configuration, local config will come before flowspec processing.
Local config will police and dscp-mark the packets and pass them to flowspec.
Flowspec will then do its processing (police, redirect) except dscp marking.

Flowspec will retain dscp marking as dictated by local qos config.

Say, we have the following:

inbound qos config : police 100Mbps, mark dscp af11
 
=============================================================
 
ipv4 access-list acl_ipv4_qos_stream
6 permit ipv4 any host 200.255.5.2
!
!
class-map match-any cm_ipv4_qos_stream
match access-group ipv4 acl_ipv4_qos_stream
 end-class-map
!
 
policy-map pm_ipv4_qos_stream
class cm_ipv4_qos_stream
  police rate 100 mbps
  !
  set dscp af11
!
 class class-default
!
 end-policy-map
!
 
interface hundredGigE 0/4/0/35
service-policy input pm_ipv4_qos_stream
 
=============================================================

Then we receive the following in flowspec advertisement.
flowspec config : police 50Mbps, mark dscp af43, redir vrf.
 
=============================================================
 
RP/0/RP0/CPU0:fretta-50#sh flowspec ipv4 detail | b 200.255.5.2
  Flow           :Dest:200.255.5.2/32
    Actions      :Traffic-rate: 50000000 bps DSCP: af43 Redirect: VRF honeypot Route-target: ASN2-4787:13  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :           116570713/12822778430        
      Transmitted         :            57360817/6309689870         
      Dropped             :            59209896/6513088560   
 
=============================================================
 
Then the outcome will be:

 

  1. traffic will be policed by flowspec at 50Mbps.
  2. traffic will be redirected by flowspec to VRF honeypot.
  3. flowspec will not overwrite dscp marking, traffic will be forwarded using dscp af11 instead of af43.

 

 


BGP Flowspec and NSR


NSR RP switchover is hitless for flowspec provided all underlay protocol (BGP, ISIS, OSPF, etc) has been configured for NSR.

 


Caveats

"hw-module profile flowspec v6-enable" config will cause both IPv6 and IPv4 linerate degradation from 835Mpps to ~700Mpps.

Flowspec processing on an ingress packet only takes place when the router does L3 lookup.
This means that flowspec won't process transit MPLS packets since it will be just a label swap instead of L3 lookup.

Put it other way, only the following ingress traffic can be processed:

  1. Plain L3 IPv4/IPv6 packets.
  2. MPLS packet with explicit-null / implicit-null label.


BGP flowspec will NOT process packets when it's received on GRE tunnel.

BGP flowspec is NOT supported on BVI interface.

BGP flowspec is NOT supported with multicast traffic.

BGP flowspec polices traffic at L2, not L1.
What this means is that the policing will not take into account L1 headers.
So, say we police at 500Mbps, then what being transmitted on the wire would be more than 500Mbps since the traffic will be transmitted with L1 headers on top of it.

Once received via BGP update, flowspec will take longer time to program to the LC if we compare with regular IPv4/IPv6 route updates.
Enhancement is being planned to make the programming faster, but it will only be provided after 6.5.1.

 

 


Related Show Commands

 

The following commands are captured from client side.

=============================================================


RP/0/RP0/CPU0:fretta-50#sh bgp ipv4 flowspec
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 7506
BGP main routing table version 7506
BGP NSR Initial initsync version 130 (Reached)
BGP NSR/ISSU Sync-Group versions 7506/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*>iDest:70.1.1.1/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
                      0.0.0.0                        10      0 ?
*>iDest:70.1.1.2/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
                      0.0.0.0                        10      0 ?
*>iDest:70.1.1.3/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
                      0.0.0.0                        10      0 ?
*>iDest:70.1.1.4/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
                      0.0.0.0                        10      0 ?
*>iDest:70.1.1.5/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
                      0.0.0.0                        10      0 ?
          
=============================================================
          
RP/0/RP0/CPU0:fretta-50#sh bgp ipv6 flowspec
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 1503
BGP main routing table version 1504
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 1504/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*>iDest:70:1:1::1/0-128,Source:80:1:1::1/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
                      202:158:2::1                  100      0 i
*>iDest:70:1:1::2/0-128,Source:80:1:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
                      202:158:2::1                  100      0 i
*>iDest:70:1:1::3/0-128,Source:80:1:1::3/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
                      202:158:2::1                  100      0 i
*>iDest:70:1:1::4/0-128,Source:80:1:1::4/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
                      202:158:2::1                  100      0 i
*>iDest:70:1:1::5/0-128,Source:80:1:1::5/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
                      202:158:2::1                  100      0 i
          
=============================================================
          
RP/0/RP0/CPU0:fretta-50#sh bgp vpnv4 flowspec
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 5
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 5/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 202.158.0.1:0 (default for vrf customer_1)
*>iDest:202.158.3.2/32,Source:202.158.1.2/32/96
                      0.0.0.0                       100      0 i
Route Distinguisher: 202.158.0.2:1
*>iDest:202.158.3.2/32,Source:202.158.1.2/32/96
                      0.0.0.0                       100      0 i

Processed 2 prefixes, 2 paths
          
=============================================================
          
RP/0/RP0/CPU0:fretta-50#sh bgp vpnv6 flowspec
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 5
BGP NSR Initial initsync version 4 (Reached)
BGP NSR/ISSU Sync-Group versions 5/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 202.158.0.1:0 (default for vrf customer_1)
*>iDest:200:158:3::2/0-128,Source:200:158:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,Length:=300,DSCP:=12/440
                      0.0.0.0                       100      0 i
Route Distinguisher: 202.158.0.2:1
*>iDest:200:158:3::2/0-128,Source:200:158:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,Length:=300,DSCP:=12/440
                      0.0.0.0                       100      0 i

Processed 2 prefixes, 2 paths
RP/0/RP0/CPU0:fretta-50#
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh bgp ipv4 flowspec summary
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 7506
BGP main routing table version 7506
BGP NSR Initial initsync version 130 (Reached)
BGP NSR/ISSU Sync-Group versions 7506/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker            7506       7506       7506       7506        7506        7506

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
200.255.1.5       0  4787    6956    2957     7506    0    0 04:47:52       1000 <-- this many flowspecs were received from server
200.255.1.6       0 50011    3015    3010        0    0    0 05:27:41  (NoNeg)
202.158.2.1       0  4787    1548    1648     7506    0    0    1d01h        250
202.158.3.1       0  4787    1683    1644     7506    0    0    1d01h        250
202.158.4.1       0  4787    1543    1649     7506    0    0    1d01h          0
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh bgp ipv6 flowspec summary
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 1503
BGP main routing table version 1504
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 1504/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker            1504       1504       1504       1504        1504        1504

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
200.255.1.5       0  4787    6957    2957     1504    0    0 04:48:02          0
200.255.1.6       0 50011    3015    3010        0    0    0 05:27:50  (NoNeg)
202.158.2.1       0  4787    1548    1648     1504    0    0    1d01h        750 <-- this many flowspecs were received from server
202.158.3.1       0  4787    1683    1644     1504    0    0    1d01h        751
202.158.4.1       0  4787    1543    1649     1504    0    0    1d01h          0
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh bgp vpnv4 flowspec summary
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 5
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 5/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker               5          5          5          5           5           5

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
202.158.2.1       0  4787    1549    1648        5    0    0    1d01h          1 <-- this many flowspecs were received from server
202.158.3.1       0  4787    1684    1644        5    0    0    1d01h          0
202.158.4.1       0  4787    1543    1649        5    0    0    1d01h          0
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh bgp vpnv6 flowspec summary
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 5
BGP NSR Initial initsync version 4 (Reached)
BGP NSR/ISSU Sync-Group versions 5/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker               5          5          5          5           5           5

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
202.158.2.1       0  4787    1549    1649        5    0    0    1d01h          1 <-- this many flowspecs were received from server
202.158.3.1       0  4787    1684    1645        5    0    0    1d01h          0
202.158.4.1       0  4787    1543    1650        5    0    0    1d01h          0
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec ipv4 detail

AFI: IPv4
  Flow           :Dest:70.1.1.1/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10
    Actions      :Traffic-rate: 0 bps  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :            18174999/3707699796         
      Transmitted         :                   0/0                  
      Dropped             :            18174999/3707699796    
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec ipv4 internal

AFI: IPv4
  Flow           :Dest:70.1.1.1/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10
    Actions      :Traffic-rate: 0 bps  (bgp.1)
      Client Version: 0
      Local:          FALSE <--- this is flowspec advertised from server
      Unsupported:    FALSE <--- flowspec data is supported
      RT:
        VRF Name Cfg:   0x00
        RT Cfg:         0x00
        RT Registered:  0x00
        RT Resolved:    0x00
    Class handles:
      Handle [0]:        30000000760013eb
    Class Handle Version:     1
    Sequence:                 1024
    Match Unsupported:        None <--- flowspec data is supported
    Synced:                   TRUE <--- flowspec is successfully synced to standby RSP (if any)
    Ref Count:                1
    Last Error:               0:Success <--- no error is seen
    Last Batch:               218
    Time Init:                May 22 11:30:13
    Time iClass Update:       May 22 11:30:13
    Statistics                        (packets/bytes)
      Matched             :            18184140/3709564560         
      Transmitted         :                   0/0                  
      Dropped             :            18184140/3709564560         
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec ipv6 detail  

AFI: IPv6
  Flow           :Dest:70:1:1::1/0-128,Source:80:1:1::1/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12
    Actions      :Traffic-rate: 1000000 bps DSCP: cs1 Nexthop: 202:158:2::1  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :            64091597/19483845488        
      Transmitted         :            33973978/10328089312        
      Dropped             :            30117619/9155756176         
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec ipv6 internal

AFI: IPv6
  Flow           :Dest:70:1:1::1/0-128,Source:80:1:1::1/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12
    Actions      :Traffic-rate: 1000000 bps DSCP: cs1 Nexthop: 202:158:2::1  (bgp.1)
      Client Version: 0
      Local:          FALSE <--- this is flowspec advertised from server
      Unsupported:    FALSE <--- flowspec data is supported
      RT:
        VRF Name Cfg:   0x00
        RT Cfg:         0x00
        RT Registered:  0x00
        RT Resolved:    0x00
    Class handles:
      Handle [0]:        30000000760005e4
    Class Handle Version:     1
    Sequence:                 1024
    Match Unsupported:        None <--- flowspec data is supported
    Synced:                   TRUE <--- flowspec is successfully synced to standby RSP (if any)
    Ref Count:                1
    Last Error:               0:Success <--- no error is seen
    Last Batch:               31
    Time Init:                May 21 15:31:39
    Time iClass Update:       May 21 15:31:39
    Statistics                        (packets/bytes)
      Matched             :            64099357/19486204528        
      Transmitted         :            33978090/10329339360        
      Dropped             :            30121267/9156865168             
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv4 detail

VRF: customer_1     AFI: IPv4
  Flow           :Dest:202.158.3.2/32,Source:202.158.1.2/32
    Actions      :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :         37260786850/4098686553500      
      Transmitted         :         21304093027/2343450232970      
      Dropped             :         15956693823/1755236320530          
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv4 internal

VRF: customer_1     AFI: IPv4
  Flow           :Dest:202.158.3.2/32,Source:202.158.1.2/32
    Actions      :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666  (bgp.1)
      Client Version: 0
      Local:          FALSE <--- this is flowspec advertised from server
      Unsupported:    FALSE <--- flowspec data is supported
      RT:
        VRF Name Cfg:   0x00
        RT Cfg:         0x01
        RT Registered:  0x01
        RT Resolved:    0x01
    Class handles:
      Handle [0]:        30000000760003ba
    Class Handle Version:     1
    Sequence:                 1024
    Match Unsupported:        None <--- flowspec data is supported
    Synced:                   TRUE <--- flowspec is successfully synced to standby RSP (if any)
    Ref Count:                1
    Last Error:               0:Success <--- no error is seen
    Last Batch:               19
    Time Init:                May 21 15:31:08
    Time iClass Update:       May 21 15:31:27
    Statistics                        (packets/bytes)
      Matched             :         37263070189/4098937720790      
      Transmitted         :         21305398659/2343593852490      
      Dropped             :         15957671530/1755343868300      
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv6 detail  

VRF: customer_1     AFI: IPv6
  Flow           :Dest:200:158:3::2/0-128,Source:200:158:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,Length:=300,DSCP:=12
    Actions      :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666  (bgp.1)
    Statistics                        (packets/bytes)
      Matched             :         16130480136/4903665961344      
      Transmitted         :          8490755776/2581189755904      
      Dropped             :          7639724360/2322476205440  
          
=============================================================
    
RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv6 internal

VRF: customer_1     AFI: IPv6
  Flow           :Dest:200:158:3::2/0-128,Source:200:158:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,Length:=300,DSCP:=12
    Actions      :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666  (bgp.1)
      Client Version: 0
      Local:          FALSE <--- this is flowspec advertised from server
      Unsupported:    FALSE <--- flowspec data is supported
      RT:
        VRF Name Cfg:   0x00
        RT Cfg:         0x01
        RT Registered:  0x01
        RT Resolved:    0x01
    Class handles:
      Handle [0]:        30000000760003bb
    Class Handle Version:     1
    Sequence:                 1024
    Match Unsupported:        None <--- flowspec data is supported
    Synced:                   TRUE <--- flowspec is successfully synced to standby RSP (if any)
    Ref Count:                1
    Last Error:               0:Success <--- no error is seen
    Last Batch:               19
    Time Init:                May 21 15:31:08
    Time iClass Update:       May 21 15:31:27
    Statistics                        (packets/bytes)
      Matched             :         16131555837/4903992974448      
      Transmitted         :          8491321864/2581361846656      
      Dropped             :          7640233973/2322631127792
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec ipv4 nlri

AFI: IPv4
  NLRI (hex)     :0x01204601010103810605815006910bb80a81c80b810a
    Actions      :Traffic-rate: 0 bps  (bgp.1)
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec ipv6 nlri

AFI: IPv6
  NLRI (hex)     :0x018000007000010001000000000000000000010280000080000100010000000000000000000103810605811606910fa00981100a91012c0b810c
    Actions      :Traffic-rate: 1000000 bps DSCP: cs1 Nexthop: 202:158:2::1  (bgp.1)
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv4 nlri

VRF: customer_1     AFI: IPv4
  NLRI (hex)     :0x0120ca9e03020220ca9e0102
    Actions      :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666  (bgp.1)
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv6 nlri

VRF: customer_1     AFI: IPv6
  NLRI (hex)     :0x018000020001580003000000000000000000020280000200015800010000000000000000000203810605811606910fa00a91012c0b810c
    Actions      :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666  (bgp.1)
          
=============================================================

RP/0/RP0/CPU0:fretta-50#sh policy-map transient type pbr                             
policy-map type pbr __bgpfs_default_IPv4
 handle:0x36000004
 table description: L3 IPv4 and IPv6
 class handle:0x760013eb  sequence 1024
   match destination-address ipv4 70.1.1.1 255.255.255.255
   match protocol tcp
   match destination-port 80
   match source-port 3000
   match packet length 200
   match dscp 10
  drop
 !
          
=============================================================

 

 

 


Logs to provide to Cisco TAC for BGP flowspec related issues on NCS5500 platform


Gather the following set of logs from client router.

Replace "NAME_OF_ROUTER"with the name of your router.

 

  1. Timestamp when the problem occurs (e.g. 16:25:15.095 GMT-7 Fri Dec 15 2017), the more exact, the better.
    It's best if the timestamp can be copied from a specific line of "show log" output.
  2. show tech flowspec file harddisk:/NAME_OF_ROUTER_sh_tech_flowspec
  3. show log | file harddisk:/NAME_OF_ROUTER_show_log.txt
    (showing the events when the problem occurs)
2 Comments

Great article!

 

Now I know how to implement flowspec in NCS5500 with arbor SP.

One question. We have NCS5504 with NC55-24H12F-SE scale-enhanced linecards. From the above I see that in 6.5.1 flowspec will be supported only for 32x100G linecards. Will our linecards be supported in the future releases, or there are any hardware limitations?

Cisco Employee

Hi @Vladimir Pisarenko, unfortunately there is no plan to support BGP flowspec feature on "NC55-24H12F-SE". This is due to hardware limitation.
BGP flowspec needs a specific type of ASIC and also external TCAM. At the time of writing, only "NC55-36X100G-A-SE" line card have all the necessary parts.
If there are future line-cards with same (or better) ASIC and external TCAM, BGP flowspec should be supported on those line-cards too.

CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards
This widget could not be displayed.