BGP routing issues on Cisco 3850

[alexnswr34]

Hi,

I've got a strange situation where a stack of 3850s (2x) is not forwarding L3 packets from directly attached networks via BGP but forwards packets from at least one hop away.
I have BGP routing running on these switches and BGP seems to work just fine.
Following are snippets of my config. There are no ACL's defined for now.

interface GigabitEthernet1/0/3
 description NEIGH_1
 no switchport
 ip address 20.30.1.2 255.255.255.252
!

interface GigabitEthernet1/0/4
 description NEIGH_2
 no switchport
 ip address 10.200.200.2 255.255.255.252
!

interface Vlan45
ip address 45.168.1.1 255.255.255.192
!
interface Vlan50
ip address 192.168.50.1 255.255.255.0

router bgp 20000
no bgp fast-external-fallover
bgp log-neighbor-changes
network 100.1.1.0 mask 255.255.255.0
network 45.168.1.0 mask 255.255.255.0
neighbor NEIGH_1 peer-group
neighbor NEIGH_1 remote-as 50000
neighbor NEIGH_1 soft-reconfiguration inbound
neighbor NEIGH_1 prefix-list DEFAULT_PUBLIC_RANGE_OUT out
neighbor NEIGH_2 peer-group
neighbor NEIGH_2 remote-as 50001
neighbor NEIGH_2 soft-reconfiguration inbound
neighbor NEIGH_2 prefix-list DEFAULT_PUBLIC_RANGE_OUT out
neighbor 20.30.1.1 peer-group NEIGH_1
neighbor 10.200.200.1 peer-group NEIGH_2

ip prefix-list DEFAULT_PUBLIC_RANGE_OUT seq 10 permit 100.1.1.0/24
ip prefix-list DEFAULT_PUBLIC_RANGE_OUT seq 20 permit 45.168.1.0/24

Default routes are provided by BGP

There is a router/firewall attached to 192.168.50.1 interface, its IP is 192.168.50.2. This router/firewall cannot get to the outside world, i.e. ping 8.8.8.8 but host on the other side of this the router with IP address of 192.168.10.10 can get to the outside world via 192.168.10.1 --> 192.168.50.1 --> 20.30.1.1 --> 8.8.8.8

Host with address 45.168.1.21 can ping 45.168.1.1 and 192.168.50.1 but not 8.8.8.8. When you do a trace route to 8.8.8.8 it shows the first hop of 45.168.1.1 then * * * *

You cannot ping 8.8.8.8 from C3850s if you issue following command "ping 8.8.8.8 source vlan45", you can ping 8.8.8.8 from C3850 is you issue following command "ping 8.8.8.8"

Switches are running Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.06.03E RELEASE SOFTWARE (fc3)

The switch MTU is set to 1998.

Any help would be much appreciated.

[Erico Verissimo]

Hi Alex,

I believe there is a firewall on your infrastructure because all IPs 192.168.50.1, 192.168.10.1, 10.30.1.2 and 10.1.1.1 are rfc 1918 and need any public IP to route on internet (NAT). If you dont have any device doing NAT, your inside address dont reach the address 8.8.8.8 because they can't be routing on the internet.

Please, if you are able to explain more your network topology it will be good to understand what is happening.

Best Regards!

Érico

[alexnswr34]

Hi Erico,

In an effort to hide real public IPs of my setup I have stuffed up the explanation of the problem itself. I have edited the original post, it reflects actual setup a bit closer now.

In addition to the above:
C3850 is a border gateway which runs BGP to a number of providers, some are external IPS type providers which are peering on public IP ranges (NEIGH_1) and some are internal data centre cross connects, those are peering on private IP ranges (NEIGH_2).

There is a public network which is directly connected to the C3850, it's IP range is 45.168.1.1/24.

There is another public network 100.1.1.0/24 which resides behind the router/firewall.

The router/firewall has following default route 0.0.0.0 0.0.0.0 192.168.50.1 - this is interface vlan50 on C3850.

C3850 has following static route:
ip route 100.1.1.0 255.255.255.0 192.168.50.2

Hots on 100.1.1.0 network can reach outside world and can be reached from the outside world OK.
The router/firewall itself cannot reach the outside world, when you do a traceroute it shows the first hop of 192.168.50.1 and then * * * *.
Hosts on 45.168.1.0 network cannot reach the outside world. When you do a traceroute it shows the first hop 45.168.1.1 followed by * * * *. Hosts on 45.168.1.0 network can reach hosts on 100.1.1.0 network via 192.168.50.0 network.

If you do a trace route to 8.8.8.8 from host with IP of 100.1.1.10 you get following.

traceroute 8.8.8.8

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets

1  100.1.1.1 (103.7.46.129)  0.447 ms  0.391 ms  0.357 ms

2  192.168.50.1 (192.168.50.1)  3.305 ms  3.611 ms  3.576 ms

3  20.30.1.1 (20.30.1.1)  0.467 ms  0.614 ms  0.594 ms

4  230.11.24.10 (230.11.24.10)  0.577 ms  0.552 ms  0.535 ms

5 102.165.240.73 (102.165.240.73)  1.460 ms  1.426 ms  1.406 ms

6  216.239.41.77 (216.239.41.77)  0.923 ms 216.239.40.223 (216.239.40.223)  0.961 ms 216.239.40.233 (216.239.40.233)  0.924 ms

7  209.85.251.53 (209.85.251.53)  0.895 ms 216.239.41.3 (216.239.41.3)  0.939 ms 209.85.244.15 (209.85.244.15)  0.897 ms

8  google-public-dns-a.google.com (8.8.8.8)  0.832 ms  0.797 ms  0.472 ms

So it seems that if the host is connected directly to the C3850 it cannot go out via BGP but if the host is at least one hop away it can use BGP routes without a problem.

[Erico Verissimo · Accepted solution]

Hi Alex,

It would be great you verify with your Service Provider whether the range 45.168.1.0/24 has any block or problem related with advertisement. I remember when I worked in a SP we have gave a public range IP and it was unblock on SP router to this range is routed on internet. It happened after this service is sold to the our customers.

When you try to ping/tracer from firewall and dont reach outside world, I think it is happening because the source IP from firewall is a private IP dont routed on internet.

Best Regard.

[alexnswr34]

OK, Figured it out with help of TAC.

Upstream provider is blocking certain networks advertised via BGP.