03-21-2024 08:16 AM - edited 03-26-2024 08:31 AM
Hello,
I have very simple Cisco EVPN VxLAN fabric running on Cisco nexus 93180YC-EX switches. I am using OSFP + iBGP for underlay/overlay with Multicast. I have border-leaf switches connected to single ISP with eBGP peers and I am advertising my public subnets to ISP.
Now I want to setup BGP Null route to stop DDoS attack. I have BGP community for RTBH from my ISP. I have configured RTBH on border leaf and try to inject null route using tag 666 but it didn't work and I didn't see it advertise any /32 route in ISP peer.
After google I came across with this doc [1] and they mentioned you have to configure RTBH on border-leaf + Remote leaf. I am confused why do I need to configure RTBH bgp community on border leaf?
This is what I have currently setup on border leaf
route-map RTBH permit 10
match tag 666
set community 1299:666
vrf context CUST
description ** VRF-CUST **
vni 10555
ip route 85.xx.xx.128/32 Null0 tag 666
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
router bgp 65001
vrf CUST
log-neighbor-changes
address-family ipv4 unicast
redistribute static route-map RTBH
aggregate-address 85.xx.xx.0/23 summary-only
aggregate-address 85.xx.xx.0/24 summary-only
aggregate-address 85.xx.xx.0/24 summary-only
I can see routes installed in table
show ip route 85.xx.xx.128 vrf CUST
IP Route Table for VRF "CUST"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
85.xx.xx.128/32, ubest/mbest: 1/0
*via Null0, [1/0], 11:17:20, static, tag 666
In BGP table
2# show ip bgp vrf CUST 85.xx.xx.128
BGP routing table information for VRF CUST, address family IPv4 Unicast
BGP routing table entry for 85.xx.xx.128/32, version 3704
Paths: (3 available, best #1)
Flags: (0x880c0042) (high32 00000000) on xmit-list, is not in urib, suppressed, exported
vpn: version 4217, (0x00000000100002) on xmit-list
Advertised path-id 1, VPN AF advertised path-id 1
Path type: redist, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path locally originated
0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.xx.1)
Origin incomplete, MED 0, localpref 100, weight 32768
Community: 1299:666
Extcommunity: RT:65001:10555
Path type: internal, path is valid, not best reason: Weight, no labeled nexthop
Imported from 10.254.110.2:32869:[2]:[0]:[0]:[48]:[d89d.6769.8da0]:[32]:[85.xx.xx.128]/272
AS-Path: NONE, path sourced internal to AS
10.255.110.0 (metric 3) from 10.254.0.1 (10.254.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10102 10555
Extcommunity: RT:65001:10102 RT:65001:10555 SOO:10.255.110.0:0 ENCAP:8
Router MAC:6cb2.aeb1.6657
Originator: 10.254.110.2 Cluster list: 10.254.0.1
Path type: internal, path is valid, not best reason: Weight, no labeled nexthop
Imported from 10.254.110.1:32869:[2]:[0]:[0]:[48]:[d89d.6769.8da0]:[32]:[85.xx.xx.128]/272
AS-Path: NONE, path sourced internal to AS
10.255.110.0 (metric 3) from 10.254.0.1 (10.254.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10102 10555
Extcommunity: RT:65001:10102 RT:65001:10555 SOO:10.255.110.0:0 ENCAP:8
Router MAC:500f.8041.ed07
Originator: 10.254.110.1 Cluster list: 10.254.0.1
VRF advertise information:
Path-id 1 not advertised to any peer
VPN AF advertise information:
Path-id 1 not advertised to any peer
That null route didn't advertised on ISP peers
# show ip bgp vrf CUST neighbors 213.xx.xx.4 advertised-routes
Peer 213.xx.xx.4 routes for address family IPv4 Unicast:
BGP table version is 3704, Local Router ID is 85.xx.xx.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
*>a85.xx.xx.0/23 0.0.0.0 100 32768 i
*>a85.xx.xx.0/24 0.0.0.0 100 32768 i
*>a85.xx.xx.0/24 0.0.0.0 100 32768 i
Just wanted to confirm with expert that do I need to configure BGP community on all my remote leaf also to get null route working?
03-21-2024 05:29 PM - edited 03-21-2024 05:55 PM
I am using route summarization do you think that could be the issue here because null route (/32) will be part of that summarization and that is why I can't see route in advertised peer?
aggregate-address 85.xx.xx.0/23 summary-only
aggregate-address 85.xx.xx.0/24 summary-only
aggregate-address 85.xx.xx.0/24 summary-only
Does BGP null route (/32) and summary-only work together?
03-21-2024 06:13 PM
Hi @satish.txt1 ,
The summary-only parameter clearly causes the host routes to be suppressed as these routes are part of the aggregate routes.
Regards,
03-21-2024 06:42 PM - edited 03-21-2024 06:48 PM
Thank for reply,
If I remove summary-only then my EVPN will blast /32 host routes to ISP routing table and I have almost ~2000 hosts in network with /32. Do you think I have to remove summary-only option in order to make BGP null route to work.
I am testing this scenario on Cisco CML lab and noticed very odd behavior that when host route /32 is installed in BGP table then BGP /32 null route isn't overriding that route and its not triggering null route.
How do I override BGP null route on existing /32 installed route?
Example: I have following /32 route installed in network.
border-1-a(config-vrf)# show ip bgp vrf ISP neighbors 101.101.101.101 advertised-routes
Peer 101.101.101.101 routes for address family IPv4 Unicast:
BGP table version is 46, Local Router ID is 69.25.124.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
*>i69.25.124.0/24 10.255.255.10 0 100 0 ?
*>i69.25.124.100/32 10.255.255.10 100 0 i
If I try to null route it won't do any effect
border-1-a(config)# vrf context ISP
border-1-a(config-vrf)# ip route 69.25.124.100/32 Null0 tag 666
border-1-a(config-vrf)# show ip bgp vrf ISP neighbors 101.101.101.101 advertised-routes
Network Next Hop Metric LocPrf Weight Path
*>i69.25.124.0/24 10.255.255.10 0 100 0 ?
*>i69.25.124.100/32 10.255.255.10 100 0 i
If I pick any random IP then it will install the route in BGP table. As you can see I have added host which doesn't exist and it tigger the community blackhole. Look like EVPN BGP required some different config to deploy null route in table.
border-1-a(config-vrf)# ip route 69.25.124.200/32 Null0 tag 666
border-1-a(config-vrf)# show ip bgp vrf ISP neighbors 101.101.101.101 advertised-routes
Network Next Hop Metric LocPrf Weight Path
*>i69.25.124.0/24 10.255.255.10 0 100 0 ?
*>i69.25.124.100/32 10.255.255.10 100 0 i
*>r69.25.124.200/32 0.0.0.0 0 100 32768 ?
.
03-22-2024 06:28 AM - edited 03-22-2024 06:30 AM
Hi @satish.txt1 ,
You can replace the "aggregate-address summary-only" with "aggregate-address suppress-map <route-map name>" to selectively suppress the more specific routes.
This allows you to only leak the more specific routes that are needed for RTBH.
Regards,
03-22-2024 06:55 AM
Hmm Interesting,
You are saying I should remove summary-only and replace aggregate-address suppress-map for all my large subnets? Sorry I never used "aggregate-address suppress-map <route-map name>" so could you please give me snippet example for my scenario.
But other problem is in EVPN host route always advertised with /32 in routing table. If I inject /32 based null route that will not get installed in table. That is what I saw in my cisco lab. May be I am missing something here.
03-22-2024 01:11 PM
Hi @satish.txt1 ,
> so could you please give me snippet example for my scenario.
aggregate-address x.x.x.x/xx suppress-map example
route-map example deny 10
match ip address prefix-list RTBH-prefixes
route-map example permit 20
ip prefix-list RTBH-prefixes seq 10 permit x.x.x.x/32
Prefixes matching the prefix-list will be denied and therefore not suppressed.
> If I inject /32 based null route that will not get installed in table.
This is not the expected behavior. The redistributed static route should be installed in the BGP table and prefer over the other BGP learnt path(s), as it is a local path. What NXOS did you run in your lab?
Regards,
03-22-2024 08:03 PM
I am testing scenario on Cisco CML lab 2.5 which comes with NXOS 10.x release. I will try your example in lab and see how it goes.
I totally agreed that locally inject static route using blackhole community should get install in route table but its not the case with my lab. you can see example here I just insert null route but you can see in table no effect
border-1-a(config)# vrf context ISP border-1-a(config-vrf)# ip route 69.25.124.100/32 Null0 tag 666 border-1-a(config-vrf)# show ip bgp vrf ISP neighbors 101.101.101.101 advertised-routes Network Next Hop Metric LocPrf Weight Path *>i69.25.124.0/24 10.255.255.10 0 100 0 ? *>i69.25.124.100/32 10.255.255.10 100 0 i
03-22-2024 10:18 PM - edited 03-22-2024 10:18 PM
Hi @satish.txt1 ,
Can you please provide the output for "sh bgp ipv4 uni 69.25.124.100/32 vrf ISP"?
Regards,
03-23-2024 07:53 AM
border-1-a# sh bgp ipv4 uni 69.25.124.100/32 vrf ISP
BGP routing table information for VRF ISP, address family IPv4 Unicast
BGP routing table entry for 69.25.124.100/32, version 98
Paths: (3 available, best #3)
Flags: (0xa8080012) (high32 00000000) on xmit-list, is in urib, is backup urib r
oute, is in HW
vpn: version 98, (0x00000000100002) on xmit-list
Path type: redist, path is valid, not best reason: MAC Mobility Sequence, no l
abeled nexthop, is extd
AS-Path: NONE, path locally originated
0.0.0.0 (metric 0) from 0.0.0.0 (69.25.124.1)
Origin incomplete, MED 0, localpref 100, weight 32768
Community: 1299:666
Extcommunity: RT:65001:10555
Path type: internal, path is valid, not best reason: Weight, no labeled nextho
p
Imported from 10.255.1.11:32867:[2]:[0]:[0]:[48]:[5254.0008.df8b]:[
32]:[69.25.124.100]/272
AS-Path: NONE, path sourced internal to AS
10.255.255.10 (metric 81) from 10.255.0.1 (10.255.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10100 10555
Extcommunity: RT:65001:10100 RT:65001:10555 SOO:10.255.255.10:0 ENCAP:8
Router MAC:5211.8d05.1b08
Originator: 10.255.1.11 Cluster list: 10.255.0.1
Advertised path-id 1, VPN AF advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop, in rib
Imported from 10.255.1.12:32867:[2]:[0]:[0]:[48]:[5254.0008.df8b]:[
32]:[69.25.124.100]/272
AS-Path: NONE, path sourced internal to AS
10.255.255.10 (metric 81) from 10.255.0.1 (10.255.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10100 10555
Extcommunity: RT:65001:10100 RT:65001:10555 SOO:10.255.255.10:0 ENCAP:8
MAC Mobility Sequence:00:1 Router MAC:520b.8335.1b08
Originator: 10.255.1.12 Cluster list: 10.255.0.1
VRF advertise information:
Path-id 1 advertised to peers:
101.101.101.101
VPN AF advertise information:
Path-id 1 not advertised to any peer
03-23-2024 10:22 AM
Hi @satish.txt1 ,
The reason the local path is not selected is that the remote path has a MAC mobility sequence extended community. This causes the remote path to be selected over the local one.
In your initial post, you included the output where the remote path does not have the MAC mobility sequence extended community and the local path is selected.
BGP routing table entry for 85.xx.xx.128/32, version 3704
Paths: (3 available, best #1)
Flags: (0x880c0042) (high32 00000000) on xmit-list, is not in urib, suppressed, exported
vpn: version 4217, (0x00000000100002) on xmit-list
Advertised path-id 1, VPN AF advertised path-id 1
Path type: redist, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path locally originated
0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.xx.1)
Origin incomplete, MED 0, localpref 100, weight 32768
Community: 1299:666
Extcommunity: RT:65001:10555
Regards,
03-23-2024 12:01 PM
Hi,
My initial post is my production network and recent one is my Cisco CML lab to mimic environment to POC.
Hmm I have to google to understand what is MAC Mobility and what is the differences between my initial post and recent one. Where is that MAC mobility coming from and is it what I need to advertise blackhole community to my ISP for null routing host? fyi, in production I do have summary-only flag set in BGP config because I can't take it out without run some test in my Cisco LAB.
I will run some test in lab and get back to you with your advice.
I really appreciate your help and thank you for taking sometime out of your day to help me.
03-24-2024 03:11 PM
Hi Harold,
Quick question, as you mentioned in my initial post I don't have MAC mobility that is why it select local path. But I can't see it in advertised peer routes because I have summary-only configured in production. If I changed summary-only with suppress-map then my problem will get resolved correct?
This is what I have in production.
vrf CUST1
log-neighbor-changes
address-family ipv4 unicast
redistribute static route-map RTBH
aggregate-address 85.xx.96.0/23 summary-only
aggregate-address 85.xx.98.0/24 summary-only
aggregate-address 85.xx.99.0/24 summary-only
Because of summary-only I can't see my null routed host IP right?
# show ip bgp vrf VIVOX neighbors 213.xxx.xxx.4 advertised-routes
Network Next Hop Metric LocPrf Weight Path
*>a85.xx.96.0/23 0.0.0.0 100 32768 i
*>a85.xx.98.0/24 0.0.0.0 100 32768 i
*>a85.xx.99.0/24 0.0.0.0 100 32768 i
But I can see in route info that my best path is locally installed prefix path
BGP routing table entry for 85.xx.xx.128/32, version 3704 Paths: (3 available, best #1) Flags: (0x880c0042) (high32 00000000) on xmit-list, is not in urib, suppressed, exported vpn: version 4217, (0x00000000100002) on xmit-list Advertised path-id 1, VPN AF advertised path-id 1 Path type: redist, path is valid, is best path, no labeled nexthop AS-Path: NONE, path locally originated 0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.xx.1) Origin incomplete, MED 0, localpref 100, weight 32768 Community: 1299:666 Extcommunity: RT:65001:10555
Am i correct in my analysis? If I use suppress-map in production setup then everything will work as I want and IP will get null route, right?
03-24-2024 06:01 PM
Hi @satish.txt1 ,
> Am i correct in my analysis? If I use suppress-map in production setup then > everything will work as I want and IP will get null route, right?
That is correct.
Regards,
03-24-2024 08:17 PM
I will try it out in my production and get back to you with result. I have one more question about your example. You have statically use /32 in prefix-list to no suppress route. I would like to have that some automated. For example I have Fastnetmon tool collecting sflow and detecting DDoS target and sending IP prefix route for null route target IP with proper community string using goBGP program. If I want to achieve kind of scenario in that case what kind of prefix list i should craft?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide