topic Hello, in Routing and SD-WAN

Problems with AWS Transit CSR, BGP and advertised routes

andrew.rogers — Tue, 05 Mar 2019 15:58:06 GMT

I started off by contacting aws support, and they gave up and told me to ask here

The idea of the AWS transit network is that it uses BGP to negotiate the interconnectivity of subnets between worldwide regions, as there is no direct way of doing so within aws

We are trying to route traffic from 172.30.0.0/16 via the transit network, and onwards to others.
TCPDump shows packets leaving this network, arriving on the destination network, and the reply being sent back. However this reply never reaches there

Some digging around on the cisco CSR showed something rather strange when looking at the bgp info

 Route Distinguisher: nnnnn:3 (default for vrf vpn-12345678)
 *>  x.x.x.x/16      y.y.y.y                         0 9059 i
 *>  x.x.x.x/16      y.y.y.y                         0 32768 ?
 *>  x.x.x.x/22      y.y.y.y         200             0 9059 i
 *>  x.x.x.x/24      y.y.y.y         200             0 9059 i
 *>  x.x.x.x/24      y.y.y.y         100             0 9059 i
 *>  x.x.x.x/24      y.y.y.y         200             0 9059 i
 *   172.30.0.0      169.254.45.17   200             0 7224 i
 *>                  y.y.y.y         100             0 7224 i
 *> x.x.x.x/22       y.y.y.y         100             0 9059 i
 *   x.x.x.x         y.y.y.y         100             0 7224 i
 *                   y.y.y.y         100             0 9059 i
 *                   y.y.y.y         100             0 9059 i
 *                   y.y.y.y         200             0 7224 i
 *>                  y.y.y.y         100             0 7224 i

I've blanked out a lot of the other details, but left the subnet masks in

My support contact at aws assured me that the vpn tunnel was sending its local network correctly, so it must be something on the cisco side of it. He recommended setting up a route map to filter out 172.30.0.0, but this does not seem to then allow 172.30.0.0/16

Has anyone got experience with manually manipulating BGP lists, or suggestions on a work around?

Hello,

Georg Pauwen — Thu, 02 Feb 2017 18:58:24 GMT

Hello,

so you are receiving the 172.30.0.0 from a Windows APIPA address (169.254.x.x)...weird indeed. Can you post the config of the CSR, and a 'show version' ?

Hello,

Georg Pauwen — Thu, 02 Feb 2017 19:07:22 GMT

Hello,

it looks like AWS is using the 169.254 for their BGP VPN peers, so the address would make sense.

The config of the CSR would be useful.

Hello

paul driver — Thu, 02 Feb 2017 19:28:33 GMT

Hello

bgp is stating even though it sees that prefix as valid its next hop isn't

With an AD of 200 I would check your ibgp peers

res

paul

Also, are you using the AWS

Georg Pauwen — Thu, 02 Feb 2017 19:44:41 GMT

Also, are you using the AWS VPC Wizard, and do you have route propagation enabled as described in the attached document:

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_VPN.html

Here's the output from show

andrew.rogers — Thu, 02 Feb 2017 19:51:32 GMT

Here's the output from show version

show version
Cisco IOS XE Software, Version 16.03.01a
Cisco IOS Software [Denali], CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.1a, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Fri 30-Sep-16 02:53 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2016 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON

ip-100-64-127-248 uptime is 7 hours, 51 minutes
Uptime for this control processor is 7 hours, 52 minutes
System returned to ROM by reload
System image file is "bootflash:packages.conf"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: ax
License Type: Default. No valid license found.
Next reload license Level: ax

cisco CSR1000V (VXE) processor (revision VXE) with 2042224K/3075K bytes of memory.
Processor board ID 9GOG3HYXLBE
1 Gigabit Ethernet interface
32768K bytes of non-volatile configuration memory.
3979824K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.
0K bytes of at webui:.

Configuration register is 0x2102

The device is an aws virtual appliance, with the cost including licensing, so am unsure why it says no valid license

I'll try and get the full config of the device done, but it will take me some time to take out sensitive info

Route propagation is enabled.

andrew.rogers — Thu, 02 Feb 2017 20:04:45 GMT

Route propagation is enabled.

AWS configures its tunnels to use 169.x.x.x addresses as the internal end points for the vpns, so the tunnel config looks like this:

interface Tunnel1
 ip vrf forwarding vpn-1234567
 ip address 169.254.45.18 255.255.255.252
 ip tcp adjust-mss 1387
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination 1.2.3.4
 tunnel protection ipsec profile ipsec-vpn-aws
 ip virtual-reassembly

and the bgp config:

router bgp 54321
 bgp log-neighbor-changes
 !
 address-family ipv4 vrf vpn-1234567
 neighbor 169.254.45.17 remote-as 7224
 neighbor 169.254.45.17 timers 10 30 30
 neighbor 169.254.45.17 activate
 neighbor 169.254.45.17 as-override
 neighbor 169.254.45.17 soft-reconfiguration inbound
 exit-address-family

I did try assigning the network 172.30.0.0 to that bgp vrf router like this:

router bgp 54321 
address-family ipv4 vrf vpn-1234567 
network 172.30.0.0 mask 255.255.0.0

But when I did a show run, it just said

network 172.30.0.0

Is it possible the show commands giving misleading information?

Hello,

Georg Pauwen — Thu, 02 Feb 2017 20:40:46 GMT

Hello,

looking at the docs from AWS, in their configuration example, they don't use a VRF. Is it possible to reconfigure and follow the guidelines as explained in the attached doc ?

http://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Cisco.html#CustomerGatewayDetail1

Using VRF is as the AWS

andrew.rogers — Fri, 03 Feb 2017 09:25:56 GMT

Using VRF is as the AWS solution causes it to be deployed. It uses various AWS technologies to detect Virtual Private Clouds that require a vpn connection. It then creates a vpn from that location to the Cisco device, parses config file and logs into the cisco devices to apply config autonomously. In theory I should never need to log into the box!

Hello Andrew,

Georg Pauwen — Fri, 03 Feb 2017 09:30:10 GMT

Hello Andrew,

makes sense. Strange though that their own docs give config examples without VRFs...

I'll look and check further...

Hello

paul driver — Fri, 03 Feb 2017 10:45:32 GMT

Hello

We are trying to route traffic from 172.30.0.0/16 via the transit network, and onwards to others.

You trying to re-advertise this /16 that originated ASN9059 to ebgp neigbour 169.254.45.17 ASN7224 as a connected route.

However it looks like ebgp neighbor 169.254.45.17 ASN7224 is already advertising this subnet towards you so you would be probably getting a conflict black holing of some traffic.

But when I did a show run, it just said
network 172.30.0.0

This is correct as its classful subnet and as such will be advertised like that.

TCPDump shows packets leaving this network, arriving on the destination network, and the reply being sent back

This is probably due to the fact ebgp neigbour 169.254.45.17 ASN7224 already has this in it rib as a connected route and thus has a rib failure for 172.30.0.0

You need to find out why that neighbor is advertising that subnet

res
Paul

I'm still none the wiser to

andrew.rogers — Fri, 03 Feb 2017 17:14:23 GMT

I'm still none the wiser to why the cisco device is showing 172.30.0.0/16 without the network mask, but I've found the cause of my routing problems

On most of my AWS route tables, I'd got the VPN gateway set to propagate the route it had going from it. On one of the tables for the destination network, this had got un-set, and not all the required manual entries had been put in. After correcting this, traffic started flowing as expected

Thanks for all the suggestions