Feature Description
Update 4/15/2021, this article will not be updated anymore going forward.
The information in this document is accurate as of IOS XR 7.0.12.
Newer version of IOS XR might have new flowspec features / enhancements that are not covered in this doc.
Whenever in doubt, please reach out to your friendly neighborhood TAC team for assistance.
Update 11/16/2021, clarifying about supported ASIC and TCAM in the "Supported Hardware" section, also putting some info for whitebox use case there.
BGP flowspec in a nutshell is a feature that will allow you to receive IPv4/IPv6 traffic flow specification (source X, destination Y, protocol UDP, source port A .. etc) and actions that need to be taken on that traffic (drop, or police .. or redirect etc) via BGP update.
Inside the BGP update, the flowspec matching criteria is represented by BGP NLRI and the actions are represented by BGP extended communities.
This feature is based on RFC 5575 and can be used to mitigate against DDoS attack. When a certain host inside of a network is being attacked, we can send a flowspec update to edge routers so that attack traffic can be policed or dropped, or even redirected elsewhere, maybe to an appliance that can clean the traffic (filter out the bad traffic and forward only the good traffic toward the affected host).
Once flowspecs have been received by a router and programmed in applicable line cards, any active L3 ports on those line cards will start processing ingress traffic according to flowspec rules.
If needed, we can disable flowspec processing on specific ports of the LC via CLI config (discussed later).
Note also that flowspec will only affect ingress traffic, it won't interfere with egress direction.
Flowspec can be programmed on different kind of interfaces:
- regular interface, e.g. TenGigE0/0/0/0
- vlan interface, e.g. TenGigE0/0/0/0.1 , Bundle-Ether3.2
- bundle interface, e.g. Bundle-Ether3
Ingress traffic can be matched by many criteria as defined in the RFC.
We can define only one, some, or all criteria to match traffic.
Once a set of criteria in a flowspec is defined, then all criteria must match the packet for actions to take place.
Available criteria:
- Type 1 - Destination Prefix
e.g. specific host 202.158.1.1/32, or maybe IP range 202.158.1.0/24 - Type 2 - Source Prefix
- Type 3 - IP Protocol
e.g. TCP, UDP
https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml - Type 4 - Port
Source or destination TCP/UDP ports, either one can match the traffic. - Type 5 - Destination port
e.g. port 80
Cisco advises to also specify "type 3 - IP protocol" matching in addition to specifying port matching.
Failure to do so might cause unintended effect such that flowspec would match other unrelated traffic. - Type 6 - Source port
e.g. port 1026
Cisco advises to also specify "type 3 - IP protocol" matching in addition to specifying port matching.
Failure to do so might cause unintended effect such that flowspec would match other unrelated traffic. - Type 7 - ICMP type
e.g. echo reply
https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol
https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol_for_IPv6 - Type 8 - ICMP code
e.g. "Destination host unreachable" code for "Destination Unreachable" type
https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol
https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol_for_IPv6 - Type 9 - TCP flags
e.g. syn
http://rapid.web.unc.edu/resources/tcp-flag-key/ - Type 10 - Packet length
Match on the total IP packet length (excluding Layer 2 but including IP header) - Type 11 - DSCP (Diffserv Code Point)
e.g. cs1
https://www.tucny.com/Home/dscp-tos - Type 12 - Fragment
e.g. MF
https://en.wikipedia.org/wiki/IPv4#Fragmentation_and_reassembly
How about a case where we receive multiple different flowspecs from controller matching the same traffic stream?
Say the controller sends us flowspec type2 (source address), type3 (protocol), and type11 (DSCP) each in different update; and later we send a single traffic stream matching all three flowspec rules.
Which flowspec will match the traffic in hardware?
Answer:
The traffic will match the flowspec with lowest numeric type value.
In this case, traffic will match flowspec type 2 (because 2<3<11).
As a result, whatever action being defined along with flowspec type 2 will be taken.
Here's an example.
Say we receive following flowspecs from controller.
RP/0/RP0/CPU0:NCS5500#sh flowspec ipv4 detail
Fri Jun 15 12:58:01.387 CEST
AFI: IPv4
Flow :Source:14.84.0.1/32
Actions :Traffic-rate: 7000000 bps (bgp.1)
Statistics (packets/bytes)
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
Flow :Proto:=255
Actions :Traffic-rate: 5000000 bps (bgp.1)
Statistics (packets/bytes)
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
Flow :DSCP:=20
Actions :Traffic-rate: 2000000 bps (bgp.1)
Statistics (packets/bytes)
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
Then we send traffic stream ingressing the router:
source 14.84.0.1
proto: 255
dscp: decimal 20 (af22)
This will result in the traffic matching flowspec type2 (source address):
RP/0/RP0/CPU0:NCS5500#sh flowspec ipv4 detail
Fri Jun 15 13:05:13.312 CEST
AFI: IPv4
Flow :Source:14.84.0.1/32
Actions :Traffic-rate: 7000000 bps (bgp.1)
Statistics (packets/bytes)
Matched : 8333868/2670601302
Transmitted : 4612813/1402458454
Dropped : 3721055/1268142848
Flow :Proto:=255
Actions :Traffic-rate: 5000000 bps (bgp.1)
Statistics (packets/bytes)
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
Flow :DSCP:=20
Actions :Traffic-rate: 2000000 bps (bgp.1)
Statistics (packets/bytes)
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
We call the router that receives the BGP update as the "client", and the router/appliance that advertises the update as the "server" or "controller".
A router than runs IOS XR can function as either client or server, and even as both server/client at the same time. For instance, when the router also needs to take actions on the attack traffic in addition to propagating the flowspec rules to clients.
BGP flowspec feature has been supported on ASR9K since XR 5.2.0, and NCS5500 supports the feature starting with 6.5.1 release.
Disclaimer:
6.5.1 is not a GA release.
The following is comparison of flowspec support between existing XR routers like ASR9K and NCS5500.
Matching criteria:
Actions:
This article will not discuss further details about working of BGP flowspec, because we already have great documentations that does that already:
1. basic description of BGP flowspec on ASR9K:
https://supportforums.cisco.com/t5/service-providers-documents/asr9000-xr-understanding-bgp-flowspec-bgp-fs/ta-p/3139916
2. detailed description of BGP flowspec on ASR9K, complete with config examples.
Please refer to CiscoLive session presented in Las Vegas (2017).
https://www.youtube.com/watch?v=dbsNf8DcNRQ
Rather, we will focus more on specific information pertaining to NCS5500 platform.
This is also a live document, we will update if new info becomes available.
Supported Hardware
If the router just needs to act as server only (no packet processing is required, perhaps the router is not in the attack path):
Any NCS5500 will do.
If the router needs to act as client (packets processing is required), we basically need platforms that is equipped with newer external TCAM.
This newer external TCAM only comes on platforms with "Jericho +" or better ASIC.
For instance, this means that flowspec is not supported with NCS-5501-SE since although comes equipped with external TCAM, it is an older TCAM version.
For client use case, two options:
- NCS5500 modular platform
Line card that RECEIVES traffic must have newer external TCAM that comes with "Jericho +" or better ASIC.
Example:
"NC55-36X100G-A-SE" (starting in 6.5.1 release)
"NC55-MOD-A-SE-S" (starting in 6.6.1 release).
Line card that TRANSMITS traffic can be of any flavor. - NCS5500 non modular platform
Same constraint as above: the platform must have newer external TCAM that comes with "Jericho +" or better ASIC.
Example:
"NCS-55A1-36H-SE-S" (starting in 6.5.1 release)
"NCS-55A2-MOD-SE-S" (starting in 6.6.1 release)
Apart from platform made by Cisco, there is also use case where IOS XR is running on 3rd party "whitebox" platform which runs with different kind of ASIC and external TCAM.
Flowspec can potentially works there but the discussion is outside of scope of this article, check with your Cisco rep for more details.
Also note that when the router acts as client, it doesn't matter either on which LC that you receive BGP update on, the LC that receives BGP update from BGP peer can be of any flavor.
Supported Scale
J+ LC: 3,000 flowspecs (a "flowspec" consists of packet-matching criteria and desired actions)
Where a single flowspec can't take more than 1,024 TCAM entries on the LC.
J2 LC: 4,000 flowspecs.
A single flowspec can take more TCAM entries than others, for instance when we define some kind of IP address range instead of a host address as matching criteria.
The scale number above applies to all use cases: IPv4 flowspec only, IPv6 flowspec only, and combined IPv4/IPv6 flowspec.
Config example
"attack traffic ingressing on interface in Global Routing Table (GRT)"
Use case:
Attack IPv4 and IPv6 traffic received on GRT interface.
Redirect attack IPv4 traffic to VRF "honeypot" for scrubbing.
Redirect attack IPv6 traffic to different NH IPv6 address in GRT for scrubbing.
Note:
- When redirecting traffic to different VRF (VRF "honeypot" in this use case), then that different VRF must have a route toward the traffic destination. Otherwise traffic will be redirected to that VRF internally but our router won't know where to forward it to.
We can add this route via static route under VRF "honeypot", as shown in following config example. - When redirecting traffic to different NH IP address and we use XR router as controller, then that controller router must have a route towards the NH IP address. Otherwise the flowspec will not be advertised to clients.
This requirement applies to both IPv4 and IPv6 flowspec.
Topology:
attack traffic
|
|
|
|
|
\/
GRT interface
NCS5500 ----------> redirect attack traffic to VRF or different NH IP address in GRT
| |
| |
| +------ iBGP ----- flowspec server
|
|
\/
attacked host
Client config
... mandatory if we want to have flowspec IPv6 support (need LC reload after commit)
... this will configure support on all supported line cards on the chassis
hw-module profile flowspec v6-enable
... or this, if only want to configure support on specific LC
hw-module profile flowspec v6-enable location <>
... activate flowspec programming in the LC
flowspec
local-install interface-all
!
... or if you want to be totally explicit:
flowspec
address-family ipv4
local-install interface-all
!
address-family ipv6
local-install interface-all
!
!
... optional, disable flowspec processing on specific ingress interfaces
interface Bundle-Ether3.1
ipv4 flowspec disable
ipv6 flowspec disable
!
... basic PASS-ALL and DROP-ALL BGP policy
route-policy PASS-ALL
pass
end-policy
!
route-policy DROP-ALL
drop
end-policy
!
... configure BGP toward flowspec server
router bgp <>
nsr
bgp router-id <>
address-family ipv4 flowspec
!
address-family ipv6 flowspec
!
neighbor <>
remote-as <>
address-family ipv4 flowspec
route-policy PASS-ALL in
route-policy DROP-ALL out
!
address-family ipv6 flowspec
route-policy PASS-ALL in
route-policy DROP-ALL out
!
update-source <>
!
!
... define VRF to redirect the attack traffic to
vrf honeypot
address-family ipv4 unicast
import route-target
4787:13
!
export route-target
4787:13
!
!
address-family ipv6 unicast
import route-target
4787:13
!
export route-target
4787:13
!
!
!
... Define static route to forward the redirected traffic under VRF.
... Here we assume the traffic destination is any hosts under 70.0.0.0/8
router static
vrf honeypot
address-family ipv4 unicast
70.0.0.0/8 200.255.55.2
!
!
!
Server Config
... basic PASS-ALL and DROP-ALL BGP policy
route-policy PASS-ALL
pass
end-policy
!
route-policy DROP-ALL
drop
end-policy
!
... configure BGP toward flowspec client
router bgp <>
address-family ipv4 flowspec
!
address-family ipv6 flowspec
!
neighbor <>
remote-as <>
address-family ipv4 flowspec
route-policy DROP-ALL in
route-policy PASS-ALL out
!
address-family ipv6 flowspec
route-policy DROP-ALL in
route-policy PASS-ALL out
!
!
!
... let's configure flowspecs to be advertised to client
...... address-family ipv4 flowspec
class-map type traffic match-all ipv4_fragment
match destination-address ipv4 70.2.1.1 255.255.255.255
match source-address ipv4 80.2.1.1 255.255.255.255
match packet length 700
match dscp af21
match fragment-type is-fragment
end-class-map
!
class-map type traffic match-all ipv4_icmp
match destination-address ipv4 70.2.1.1 255.255.255.255
match source-address ipv4 80.2.1.1 255.255.255.255
match packet length 700
match dscp af21
match fragment-type is-fragment
match ipv4 icmp-type 3
match ipv4 icmp-code 2
end-class-map
!
policy-map type pbr scale_ipv4
class type traffic ipv4_fragment
drop
!
class type traffic ipv4_icmp
police rate 1 mbps
!
set dscp cs2
redirect nexthop route-target 4787:13
!
class type traffic class-default
!
end-policy-map
!
flowspec
address-family ipv4
service-policy type pbr scale_ipv4
!
!
...... address-family ipv6 flowspec
class-map type traffic match-all ipv6_tcp
match destination-address ipv6 70:1:1::5a/128
match source-address ipv6 80:1:1::5a/128
match protocol tcp
match destination-port 22
match source-port 4000
match tcp-flag 0x10
match packet length 300
match dscp af12
end-class-map
!
class-map type traffic match-all ipv6_icmp
match destination-address ipv6 70:2:1::1/128
match source-address ipv6 80:2:1::1/128
match packet length 800
match dscp af22
match ipv6 icmp-type 4
match ipv6 icmp-code 1
end-class-map
!
policy-map type pbr scale_ipv6
class type traffic ipv6_tcp
police rate 1 mbps
!
set dscp cs1
redirect ipv6 nexthop 202:158:2::1
!
class type traffic ipv6_icmp
police rate 1 mbps
!
set dscp cs3
redirect nexthop route-target 4787:13
!
class type traffic class-default
!
!
flowspec
address-family ipv6
service-policy type pbr scale_ipv6
!
!
Config example
"attack traffic ingressing on VRF interface"
Use case:
Attack traffic received on VRF "customer_1" interface.
Redirect this traffic to VRF "dirty-dancing" for scrubbing.
Note:
If we're redirecting traffic to different VRF (VRF "dirty-dancing" in this use case), then that different VRF must have a route toward the traffic destination. Otherwise traffic will be redirected to that VRF internally but our router won't know where to forward it to.
In this usecase, we will forward the traffic under VRF "dirty-dancing" using L3VPN route.
Our router gets this route from another L3VPN PE that is connected to redirect destination, so we won't need to define static routing under VRF like previous use case.
Topology:
attack traffic
|
|
|
|
|
\/
VRF "customer_1" interface
NCS5500 ----------> redirect attack traffic to different VRF "dirty-dancing"
| |
| |
| +------ iBGP ----- flowspec server
|
|
\/
attacked host (VRF "customer_1")
Client Config
... mandatory if we want to have flowspec IPv6 support (need LC reload after commit)
... this will configure support on all supported line cards on the chassis
hw-module profile flowspec v6-enable
... or this, if only want to configure support on specific LC
hw-module profile flowspec v6-enable location <>
... activate flowspec programming in the LC
flowspec
local-install interface-all
!
... or if you want to be totally explicit:
flowspec
vrf customer_1
address-family ipv4
local-install interface-all
!
address-family ipv6
local-install interface-all
!
!
!
... optional, disable flowspec processing on specific ingress interfaces
interface Bundle-Ether3.1
ipv4 flowspec disable
ipv6 flowspec disable
!
... basic PASS-ALL and DROP-ALL BGP policy
route-policy PASS-ALL
pass
end-policy
!
route-policy DROP-ALL
drop
end-policy
!
... configure BGP toward flowspec server
router bgp <>
nsr
bgp router-id <>
address-family vpnv4 unicast
!
address-family vpnv6 unicast
!
address-family vpnv4 flowspec
!
address-family vpnv6 flowspec
!
neighbor <>
remote-as <>
address-family vpnv4 unicast
route-policy PASS-ALL in
route-policy PASS-ALL out
!
address-family vpnv6 unicast
route-policy PASS-ALL in
route-policy PASS-ALL out
!
address-family vpnv4 flowspec
route-policy PASS-ALL in
route-policy DROP-ALL out
!
address-family vpnv6 flowspec
route-policy PASS-ALL in
route-policy DROP-ALL out
!
update-source <>
!
vrf customer_1
rd auto
address-family ipv4 unicast
label mode per-vrf
redistribute connected
!
address-family ipv6 unicast
label mode per-vrf
redistribute connected
!
address-family ipv4 flowspec
!
address-family ipv6 flowspec
!
!
vrf dirty_dancing
rd auto
address-family ipv4 unicast
label mode per-vrf
!
address-family ipv6 unicast
label mode per-vrf
!
!
!
... define VRF "customer_1" and "dirty_dancing"
vrf customer_1
address-family ipv4 unicast
import route-target
4787:1313
!
export route-target
4787:1313
!
!
address-family ipv4 flowspec
import route-target
4787:1313
!
export route-target
4787:1313
!
!
address-family ipv6 unicast
import route-target
4787:1313
!
export route-target
4787:1313
!
!
address-family ipv6 flowspec
import route-target
4787:1313
!
export route-target
4787:1313
!
!
!
vrf dirty_dancing
address-family ipv4 unicast
import route-target
4787:666
!
export route-target
4787:666
!
!
address-family ipv6 unicast
import route-target
4787:666
!
export route-target
4787:666
!
!
!
Server Config
... basic PASS-ALL and DROP-ALL BGP policy
route-policy PASS-ALL
pass
end-policy
!
route-policy DROP-ALL
drop
end-policy
!
... configure BGP toward flowspec client
router bgp <>
nsr
bgp router-id <>
address-family vpnv4 unicast
!
address-family vpnv6 unicast
!
address-family vpnv4 flowspec
!
address-family vpnv6 flowspec
!
neighbor <>
remote-as <>
address-family vpnv4 unicast
route-policy PASS-ALL in
route-policy PASS-ALL out
!
address-family vpnv6 unicast
route-policy PASS-ALL in
route-policy PASS-ALL out
!
address-family vpnv4 flowspec
route-policy DROP-ALL in
route-policy PASS-ALL out
!
address-family vpnv6 flowspec
route-policy DROP-ALL in
route-policy PASS-ALL out
!
update-source <>
!
vrf customer_1
rd auto
address-family ipv4 unicast
label mode per-vrf
redistribute connected
!
address-family ipv6 unicast
label mode per-vrf
redistribute connected
!
address-family ipv4 flowspec
!
address-family ipv6 flowspec
!
!
!
... let's configure flowspecs to be advertised to client
Note:
For a flowspec that is configured under VRF like the following, we can only use "redirect nexthop route-target <>" on the controller side to redirect this traffic to other VRF.
"redirect ipv4|ipv6 nexthop <>" is not supported in a VRF context.
...... address-family vpnv4 flowspec
class-map type traffic match-all l3vpn_ipv4_attack
match destination-address ipv4 202.158.3.2 255.255.255.255
match source-address ipv4 202.158.1.2 255.255.255.255
end-class-map
!
policy-map type pbr pm_cust_VRF_to_diff_VRF
class type traffic l3vpn_ipv4_attack
redirect nexthop route-target 4787:666
set dscp cs6
police rate 250 mbps
!
class type traffic class-default
!
end-policy-map
!
flowspec
vrf customer_1
address-family ipv4
service-policy type pbr pm_cust_VRF_to_diff_VRF
!
!
...... address-family vpnv6 flowspec
class-map type traffic match-all l3vpn_ipv6_attack
match destination-address ipv6 200:158:3::2/128
match source-address ipv6 200:158:1::2/128
match protocol tcp
match destination-port 22
match source-port 4000
match packet length 300
match dscp af12
end-class-map
!
policy-map type pbr pm_cust_VRF_to_diff_VRF_ipv6
class type traffic l3vpn_ipv6_attack
redirect nexthop route-target 4787:666
set dscp cs6
police rate 250 mbps
!
class type traffic class-default
!
end-policy-map
!
flowspec
vrf customer_1
address-family ipv6
service-policy type pbr pm_cust_VRF_to_diff_VRF_ipv6
!
!
Config example
XR platforms as flowspec controllers
More detailed config examples to define packet matching criteria (class-maps) when using XR platforms as flowspec controller can be found here:
https://supportforums.cisco.com/t5/service-providers-blogs/ncs5500-bgp-flowspec-packet-matching-criteria/ba-p/3387457
BGP Flowspec and IPv6 BGP neighbors
The use-cases and config examples provided previously are when we have BGP neighbors that runs on IPv4.
What about if we have IPv6 BGP neighbors instead of IPv4?
We can still advertise flowspec to that neighbor, but only for address-family "ipv6 flowspec".
The following flowspec address-families are not supported:
- ipv4 flowspec
- vpnv4 flowspec
- vpnv6 flowspec
This is by design and same limitation applies to other platforms that run IOS-XR like ASR9000.
BGP Flowspec and bundle interface
As mentioned before, BGP flowspec is also supported on bundle interface.
Note the following bundle behavior when it comes to flowspec traffic policing, however.
Say, we have interface Bundle-Ether3 with member links Hu0/4/0/0 and Hu0/4/0/9.
The member links are served by different NPU on LC 0/4/CPU0.
Hu0/4/0/0 will be served by NPU 0.
Hu0/4/0/9 will be served by NPU 1.
RP/0/RP0/CPU0:fretta-50#sh controllers npu voq-usage interface Hu0/4/0/0 instance all location 0/4/cpu0
-------------------------------------------------------------------
Node ID: 0/4/CPU0
Intf Intf NPU NPU PP Sys VOQ Flow VOQ Port
name handle # core Port Port base base port speed
(hex) type
----------------------------------------------------------------------
Hu0/4/0/0 2000130 0 1 21 2421 1024 4312 local 100G
RP/0/RP0/CPU0:fretta-50#
RP/0/RP0/CPU0:fretta-50#sh controllers npu voq-usage interface Hu0/4/0/9 instance all location 0/4/cpu0
-------------------------------------------------------------------
Node ID: 0/4/CPU0
Intf Intf NPU NPU PP Sys VOQ Flow VOQ Port
name handle # core Port Port base base port speed
(hex) type
----------------------------------------------------------------------
Hu0/4/0/9 2000178 1 1 21 2521 1096 4312 local 100G
And let's say there is currently a flowspec being advertised to our router:
policy-map type pbr scale_ipv4
class type traffic ipv4_icmp
police rate 1 mbps
!
!
class type traffic class-default
!
end-policy-map
!
Now what's going to happen is that the police action will be programmed to both NPU 0 and NPU 1.
Hu0/4/0/0 on NPU 0 will police at 1Mbps.
Hu0/4/0/9 on NPU 1 will police at 1Mbps.
Assuming we receive many ICMP streams such that the streams will be load-balanced pretty evenly on both member links, this will give you 2Mbps police rate total instead of 1Mbps.
BGP Flowspec and BGP route-policy
We can configure BGP route-policy to filter flowspec updates.
One use case is maybe to limit flowspec updates with only specific subnet masks.
For instance:
router bgp <>
!
address-family ipv4 flowspec
!
address-family ipv6 flowspec
!
neighbor <>
remote-as <>
update-source <>
address-family ipv4 flowspec
route-policy routes_flowspec in
route-policy drop-all out
!
...
route-policy routes_flowspec
if destination in ge_17 then
pass
endif
end-policy
prefix-set ge_17
0.0.0.0/0 ge 17
end-set
!
Just be careful when doing route-policy.
If you commit the above config, and the coming flowspec updates don't have Type 1 (Destination address) criteria, then no updates would match the policy and it would be implicit deny for all flowspec updates.
BGP Flowspec and ACL
When flowspec is implemented on an interface that is also having ingress ACL, ACL will come before flowspec processing.
- When the ACL is permitting the traffic:
Flowspec will take whatever passed by ACL and run flowspec processing. - When the ACL is denying the traffic:
Flowspec will not process any traffic since ACL has discarded them.
BGP Flowspec and local QoS configuration
When flowspec is implemented on an interface that is also having local QoS configuration, local config will come before flowspec processing.
Local config will police and dscp-mark the packets and pass them to flowspec.
Flowspec will then do its processing (police, redirect) except dscp marking.
Flowspec will retain dscp marking as dictated by local qos config.
Say, we have the following:
inbound qos config : police 100Mbps, mark dscp af11
=============================================================
ipv4 access-list acl_ipv4_qos_stream
6 permit ipv4 any host 200.255.5.2
!
!
class-map match-any cm_ipv4_qos_stream
match access-group ipv4 acl_ipv4_qos_stream
end-class-map
!
policy-map pm_ipv4_qos_stream
class cm_ipv4_qos_stream
police rate 100 mbps
!
set dscp af11
!
class class-default
!
end-policy-map
!
interface hundredGigE 0/4/0/35
service-policy input pm_ipv4_qos_stream
=============================================================
Then we receive the following in flowspec advertisement.
flowspec config : police 50Mbps, mark dscp af43, redir vrf.
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec ipv4 detail | b 200.255.5.2
Flow :Dest:200.255.5.2/32
Actions :Traffic-rate: 50000000 bps DSCP: af43 Redirect: VRF honeypot Route-target: ASN2-4787:13 (bgp.1)
Statistics (packets/bytes)
Matched : 116570713/12822778430
Transmitted : 57360817/6309689870
Dropped : 59209896/6513088560
=============================================================
Then the outcome will be:
- traffic will be policed by flowspec at 50Mbps.
- traffic will be redirected by flowspec to VRF honeypot.
- flowspec will not overwrite dscp marking, traffic will be forwarded using dscp af11 instead of af43.
BGP Flowspec and NSR
NSR RP switchover is hitless for flowspec provided all underlay protocol (BGP, ISIS, OSPF, etc) has been configured for NSR.
Caveats
"hw-module profile flowspec v6-enable" config will cause both IPv6 and IPv4 linerate degradation from 835Mpps to ~700Mpps.
Flowspec processing on an ingress packet only takes place when the router does L3 lookup.
This means that flowspec won't process transit MPLS packets since it will be just a label swap instead of L3 lookup.
Put it other way, only the following ingress traffic can be processed:
- Plain L3 IPv4/IPv6 packets.
- MPLS (labeled) packet in which the router is an MPLS termination point, such that all labels will be popped and L3 lookup will take place.
Flowspec is capable to process stacked labels traffic (such that L3VPN packets that have VPN label plus next-hop label) or a single label traffic (explicit null), again as long as L3 lookup takes place.
Flowspec is only active on a VRF (whether it's a default VRF or custom VRF) only if there's at least one interface active with the right address-family within that VRF. This caveat won't affect most of flowspec use cases, but there are some use cases where this comes into play:
1. Say you have an IPv6 L3VPN over an IPv4 only MPLS core, then when an IPv6 L3VPN traffic arrive at PE egress router, flowspec can't do anything about this traffic.
Why? Because all of the default VRF interfaces (core facing interfaces) are not configured with IPv6 address.
2. Say you receive a traffic on a custom VRF with no interfaces configured (since next hop will resolve to different VRF via route leaking), then flowspec can't do anything about this traffic either.
Why? Because there are no interface configured with this VRF.
The work around for these two cases is to configure a "dummy" VLAN interface that belongs to the ingress VRF (default VRF for use case 1, custom VRF for use case 2) with right address-family (IPv4 or IPv6). This VLAN interface can be put in shutdown state for security purpose.
BGP flowspec will NOT process packets when it's received on GRE tunnel.
BGP flowspec is NOT supported on BVI interface.
BGP flowspec is NOT supported with multicast traffic.
BGP flowspec polices traffic at L2, not L1.
What this means is that the policing will not take into account L1 headers.
So, say we police at 500Mbps, then what being transmitted on the wire would be more than 500Mbps since the traffic will be transmitted with L1 headers on top of it.
Once received via BGP update, flowspec will take longer time to program to the LC if we compare with regular IPv4/IPv6 route updates.
Enhancement is being planned to make the programming faster, but it will only be provided after 6.5.1.
Related Show Commands
The following commands are captured from client side.
=============================================================
RP/0/RP0/CPU0:fretta-50#sh bgp ipv4 flowspec
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 7506
BGP main routing table version 7506
BGP NSR Initial initsync version 130 (Reached)
BGP NSR/ISSU Sync-Group versions 7506/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>iDest:70.1.1.1/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
0.0.0.0 10 0 ?
*>iDest:70.1.1.2/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
0.0.0.0 10 0 ?
*>iDest:70.1.1.3/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
0.0.0.0 10 0 ?
*>iDest:70.1.1.4/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
0.0.0.0 10 0 ?
*>iDest:70.1.1.5/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10/176
0.0.0.0 10 0 ?
=============================================================
RP/0/RP0/CPU0:fretta-50#sh bgp ipv6 flowspec
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 1503
BGP main routing table version 1504
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 1504/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>iDest:70:1:1::1/0-128,Source:80:1:1::1/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
202:158:2::1 100 0 i
*>iDest:70:1:1::2/0-128,Source:80:1:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
202:158:2::1 100 0 i
*>iDest:70:1:1::3/0-128,Source:80:1:1::3/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
202:158:2::1 100 0 i
*>iDest:70:1:1::4/0-128,Source:80:1:1::4/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
202:158:2::1 100 0 i
*>iDest:70:1:1::5/0-128,Source:80:1:1::5/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12/464
202:158:2::1 100 0 i
=============================================================
RP/0/RP0/CPU0:fretta-50#sh bgp vpnv4 flowspec
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 5
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 5/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 202.158.0.1:0 (default for vrf customer_1)
*>iDest:202.158.3.2/32,Source:202.158.1.2/32/96
0.0.0.0 100 0 i
Route Distinguisher: 202.158.0.2:1
*>iDest:202.158.3.2/32,Source:202.158.1.2/32/96
0.0.0.0 100 0 i
Processed 2 prefixes, 2 paths
=============================================================
RP/0/RP0/CPU0:fretta-50#sh bgp vpnv6 flowspec
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 5
BGP NSR Initial initsync version 4 (Reached)
BGP NSR/ISSU Sync-Group versions 5/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 202.158.0.1:0 (default for vrf customer_1)
*>iDest:200:158:3::2/0-128,Source:200:158:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,Length:=300,DSCP:=12/440
0.0.0.0 100 0 i
Route Distinguisher: 202.158.0.2:1
*>iDest:200:158:3::2/0-128,Source:200:158:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,Length:=300,DSCP:=12/440
0.0.0.0 100 0 i
Processed 2 prefixes, 2 paths
RP/0/RP0/CPU0:fretta-50#
=============================================================
RP/0/RP0/CPU0:fretta-50#sh bgp ipv4 flowspec summary
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 7506
BGP main routing table version 7506
BGP NSR Initial initsync version 130 (Reached)
BGP NSR/ISSU Sync-Group versions 7506/0
BGP scan interval 60 secs
BGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 7506 7506 7506 7506 7506 7506
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
200.255.1.5 0 4787 6956 2957 7506 0 0 04:47:52 1000 <-- this many flowspecs were received from server
200.255.1.6 0 50011 3015 3010 0 0 0 05:27:41 (NoNeg)
202.158.2.1 0 4787 1548 1648 7506 0 0 1d01h 250
202.158.3.1 0 4787 1683 1644 7506 0 0 1d01h 250
202.158.4.1 0 4787 1543 1649 7506 0 0 1d01h 0
=============================================================
RP/0/RP0/CPU0:fretta-50#sh bgp ipv6 flowspec summary
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 1503
BGP main routing table version 1504
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 1504/0
BGP scan interval 60 secs
BGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 1504 1504 1504 1504 1504 1504
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
200.255.1.5 0 4787 6957 2957 1504 0 0 04:48:02 0
200.255.1.6 0 50011 3015 3010 0 0 0 05:27:50 (NoNeg)
202.158.2.1 0 4787 1548 1648 1504 0 0 1d01h 750 <-- this many flowspecs were received from server
202.158.3.1 0 4787 1683 1644 1504 0 0 1d01h 751
202.158.4.1 0 4787 1543 1649 1504 0 0 1d01h 0
=============================================================
RP/0/RP0/CPU0:fretta-50#sh bgp vpnv4 flowspec summary
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 5
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 5/0
BGP scan interval 60 secs
BGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 5 5 5 5 5 5
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
202.158.2.1 0 4787 1549 1648 5 0 0 1d01h 1 <-- this many flowspecs were received from server
202.158.3.1 0 4787 1684 1644 5 0 0 1d01h 0
202.158.4.1 0 4787 1543 1649 5 0 0 1d01h 0
=============================================================
RP/0/RP0/CPU0:fretta-50#sh bgp vpnv6 flowspec summary
BGP router identifier 202.158.0.1, local AS number 4787
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 5
BGP NSR Initial initsync version 4 (Reached)
BGP NSR/ISSU Sync-Group versions 5/0
BGP scan interval 60 secs
BGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 5 5 5 5 5 5
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
202.158.2.1 0 4787 1549 1649 5 0 0 1d01h 1 <-- this many flowspecs were received from server
202.158.3.1 0 4787 1684 1645 5 0 0 1d01h 0
202.158.4.1 0 4787 1543 1650 5 0 0 1d01h 0
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec ipv4 detail
AFI: IPv4
Flow :Dest:70.1.1.1/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10
Actions :Traffic-rate: 0 bps (bgp.1)
Statistics (packets/bytes)
Matched : 18174999/3707699796
Transmitted : 0/0
Dropped : 18174999/3707699796
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec ipv4 internal
AFI: IPv4
Flow :Dest:70.1.1.1/32,Proto:=6,DPort:=80,SPort:=3000,Length:=200,DSCP:=10
Actions :Traffic-rate: 0 bps (bgp.1)
Client Version: 0
Local: FALSE <--- this is flowspec advertised from server
Unsupported: FALSE <--- flowspec data is supported
RT:
VRF Name Cfg: 0x00
RT Cfg: 0x00
RT Registered: 0x00
RT Resolved: 0x00
Class handles:
Handle [0]: 30000000760013eb
Class Handle Version: 1
Sequence: 1024
Match Unsupported: None <--- flowspec data is supported
Synced: TRUE <--- flowspec is successfully synced to standby RSP (if any)
Ref Count: 1
Last Error: 0:Success <--- no error is seen
Last Batch: 218
Time Init: May 22 11:30:13
Time iClass Update: May 22 11:30:13
Statistics (packets/bytes)
Matched : 18184140/3709564560
Transmitted : 0/0
Dropped : 18184140/3709564560
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec ipv6 detail
AFI: IPv6
Flow :Dest:70:1:1::1/0-128,Source:80:1:1::1/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12
Actions :Traffic-rate: 1000000 bps DSCP: cs1 Nexthop: 202:158:2::1 (bgp.1)
Statistics (packets/bytes)
Matched : 64091597/19483845488
Transmitted : 33973978/10328089312
Dropped : 30117619/9155756176
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec ipv6 internal
AFI: IPv6
Flow :Dest:70:1:1::1/0-128,Source:80:1:1::1/0-128,NH:=6,DPort:=22,SPort:=4000,TCPFlags:=0x10,Length:=300,DSCP:=12
Actions :Traffic-rate: 1000000 bps DSCP: cs1 Nexthop: 202:158:2::1 (bgp.1)
Client Version: 0
Local: FALSE <--- this is flowspec advertised from server
Unsupported: FALSE <--- flowspec data is supported
RT:
VRF Name Cfg: 0x00
RT Cfg: 0x00
RT Registered: 0x00
RT Resolved: 0x00
Class handles:
Handle [0]: 30000000760005e4
Class Handle Version: 1
Sequence: 1024
Match Unsupported: None <--- flowspec data is supported
Synced: TRUE <--- flowspec is successfully synced to standby RSP (if any)
Ref Count: 1
Last Error: 0:Success <--- no error is seen
Last Batch: 31
Time Init: May 21 15:31:39
Time iClass Update: May 21 15:31:39
Statistics (packets/bytes)
Matched : 64099357/19486204528
Transmitted : 33978090/10329339360
Dropped : 30121267/9156865168
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv4 detail
VRF: customer_1 AFI: IPv4
Flow :Dest:202.158.3.2/32,Source:202.158.1.2/32
Actions :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666 (bgp.1)
Statistics (packets/bytes)
Matched : 37260786850/4098686553500
Transmitted : 21304093027/2343450232970
Dropped : 15956693823/1755236320530
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv4 internal
VRF: customer_1 AFI: IPv4
Flow :Dest:202.158.3.2/32,Source:202.158.1.2/32
Actions :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666 (bgp.1)
Client Version: 0
Local: FALSE <--- this is flowspec advertised from server
Unsupported: FALSE <--- flowspec data is supported
RT:
VRF Name Cfg: 0x00
RT Cfg: 0x01
RT Registered: 0x01
RT Resolved: 0x01
Class handles:
Handle [0]: 30000000760003ba
Class Handle Version: 1
Sequence: 1024
Match Unsupported: None <--- flowspec data is supported
Synced: TRUE <--- flowspec is successfully synced to standby RSP (if any)
Ref Count: 1
Last Error: 0:Success <--- no error is seen
Last Batch: 19
Time Init: May 21 15:31:08
Time iClass Update: May 21 15:31:27
Statistics (packets/bytes)
Matched : 37263070189/4098937720790
Transmitted : 21305398659/2343593852490
Dropped : 15957671530/1755343868300
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv6 detail
VRF: customer_1 AFI: IPv6
Flow :Dest:200:158:3::2/0-128,Source:200:158:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,Length:=300,DSCP:=12
Actions :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666 (bgp.1)
Statistics (packets/bytes)
Matched : 16130480136/4903665961344
Transmitted : 8490755776/2581189755904
Dropped : 7639724360/2322476205440
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv6 internal
VRF: customer_1 AFI: IPv6
Flow :Dest:200:158:3::2/0-128,Source:200:158:1::2/0-128,NH:=6,DPort:=22,SPort:=4000,Length:=300,DSCP:=12
Actions :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666 (bgp.1)
Client Version: 0
Local: FALSE <--- this is flowspec advertised from server
Unsupported: FALSE <--- flowspec data is supported
RT:
VRF Name Cfg: 0x00
RT Cfg: 0x01
RT Registered: 0x01
RT Resolved: 0x01
Class handles:
Handle [0]: 30000000760003bb
Class Handle Version: 1
Sequence: 1024
Match Unsupported: None <--- flowspec data is supported
Synced: TRUE <--- flowspec is successfully synced to standby RSP (if any)
Ref Count: 1
Last Error: 0:Success <--- no error is seen
Last Batch: 19
Time Init: May 21 15:31:08
Time iClass Update: May 21 15:31:27
Statistics (packets/bytes)
Matched : 16131555837/4903992974448
Transmitted : 8491321864/2581361846656
Dropped : 7640233973/2322631127792
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec ipv4 nlri
AFI: IPv4
NLRI (hex) :0x01204601010103810605815006910bb80a81c80b810a
Actions :Traffic-rate: 0 bps (bgp.1)
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec ipv6 nlri
AFI: IPv6
NLRI (hex) :0x018000007000010001000000000000000000010280000080000100010000000000000000000103810605811606910fa00981100a91012c0b810c
Actions :Traffic-rate: 1000000 bps DSCP: cs1 Nexthop: 202:158:2::1 (bgp.1)
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv4 nlri
VRF: customer_1 AFI: IPv4
NLRI (hex) :0x0120ca9e03020220ca9e0102
Actions :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666 (bgp.1)
=============================================================
RP/0/RP0/CPU0:fretta-50#sh flowspec vrf customer_1 ipv6 nlri
VRF: customer_1 AFI: IPv6
NLRI (hex) :0x018000020001580003000000000000000000020280000200015800010000000000000000000203810605811606910fa00a91012c0b810c
Actions :Traffic-rate: 250000000 bps DSCP: cs6 Redirect: VRF dirty_dancing Route-target: ASN2-4787:666 (bgp.1)
=============================================================
RP/0/RP0/CPU0:fretta-50#sh policy-map transient type pbr
policy-map type pbr __bgpfs_default_IPv4
handle:0x36000004
table description: L3 IPv4 and IPv6
class handle:0x760013eb sequence 1024
match destination-address ipv4 70.1.1.1 255.255.255.255
match protocol tcp
match destination-port 80
match source-port 3000
match packet length 200
match dscp 10
drop
!
=============================================================
Logs to provide to Cisco TAC for BGP flowspec related issues on NCS5500 platform
Gather the following set of logs from client router.
Replace "NAME_OF_ROUTER"with the name of your router.
Generic logs to grab:
- Timestamp when the problem occurs (e.g. 16:25:15.095 GMT-7 Fri Dec 15 2017), the more exact, the better.
It's best if the timestamp can be copied from a specific line of "show log" output. - show tech flowspec file harddisk:/NAME_OF_ROUTER_sh_tech_flowspec
- show controller fia trace all location <ingress LC> | file harddisk:/NAME_OF_ROUTER_show_controller_fia_trace_all_location_ingress_lc.txt
- show ofa trace loca <ingress LC> | file harddisk:/NAME_OF_ROUTER_sh_ofa_trace_location_ingress_lc.txt
- show log | file harddisk:/NAME_OF_ROUTER_show_log.txt
(showing the events when the problem occurs)
If using Jericho2 LC, grab following logs too:
- sh controllers fia diagshell <NPU ID> "dbal tbl du tbl=v6_fs" location <ingress LC> | file harddisk:/NAME_OF_ROUTER_show_controller_fia_diagshell_npu_dbal_tbl_v6_fs.txt
- sh controllers fia diagshell <NPU ID> "dbal tbl du tbl=v4_fs" location <ingress LC> | file harddisk:/NAME_OF_ROUTER_show_controller_fia_diagshell_npu_dbal_tbl_v4_fs.txt
- sh controllers fia diagshell <NPU ID> "diag count g" location <ingress LC> | file harddisk:/NAME_OF_ROUTER_show_controller_fia_diagshell_npu_diag_count_g.txt
- sh controllers fia diagshell <NPU ID> "pp vis ikleap" location <ingress LC> | file harddisk:/NAME_OF_ROUTER_show_controller_fia_diagshell_npu_pp_vis_ikleap.txt
3 Comments