Unexpected route

[BobGreer65666]

Hi there,

Thanks for reading.

This is an extension of a recent query I had: IP SLA - TRACKING - ISPs.

I have a new Internet circuit for my small network.  I have two hub sites in the private LAN with a core switch at each location.  The main site Exec core switch has a 0.0.0.0/0 pointing to the Exec FW.  When I do the same at my other hub site (0.0.0.0/0 pointing to NWI FW) pings to the Internet fail.  I've tried pinging from different vrf sources.

Running packet-tracer on the FTD firewall shows the traffic is allowed.  That is expected - the Inside interface has a permit any any from the Inside LAN to the outside world.

I think I have a hole in my routing logic.

Attached are a high level diagram and what I think are the important routing outputs.

Thanks again for reading!

Bob

@ Georg & Balaji: again, thanks for your help on the IP SLA - I saw the routes being removed and added when I shut/no-shut the appropriate testing interface.

[Georg Pauwen]

Hello,

there are no attachments ?

[BobGreer65666]

Hi Georg,

Once more, with attachments this time.

[Giuseppe Larosa]

Hello @BobGreer65666 ,

>> The main site Exec core switch has a 0.0.0.0/0 pointing to the Exec FW. When I do the same at my other hub site (0.0.0.0/0 pointing to NWI FW) pings to the Internet fail. I've tried pinging from different vrf sources.

We cannot see your network topology, however we can say you need to create a clear hierarchy for defaul routes:

both hub routers / switches should point to the SAME FW and eventually have the other FW as a backup floating route

With firewalls you need to perform symmetric routing that is: for a flow with IP address A.A.A.A and destination E.E.E.E the return path must be via the SAME firewall that is sending in the outgoing direction ( to the internet).

This is because firewalls are stateful and need to see the return traffic on the same FW that has started the inside to outside session . If the packets are sent to a FW for example NMWI and return traffic arrives at the other FW Exec FW the return traffic is dropped.

Hope to help

Giuseppe

[BobGreer65666]

H8 when I do that....

Thanks for looking into this

[BobGreer65666]

Hi Giuseppe,

In my testing, I shut the interface connecting to the primary egress so there was only the secondary WAN link available.  Return traffic from the outside should have been guaranteed to come back to the secondary link.

Are you suspecting that the return traffic doesn't know how to find its way back once inside the failover egress?  I was seeing expired TTLs so for sure a routing issue.  I think packets were expiring between the two cores.  Does that mean anything to you?

Thanks again for reading!

[Richard Burts]

When I read a post that describes pings to the Internet fail, the first thing that I think of is the possibility of problems with address translation. Do both firewalls have address translation configured for all of the inside subnets?

You tell us "I think packets were expiring between the two cores" If packets are expiring between the cores it sounds like there might be a loop. In looking at the posted commands I see that the default route has the next hop as 10.0.0.2 in vlan 10. Is it possible that 10.0.0.2 has a default route pointing at 10.0.0.3? 

[BobGreer65666]

Hi Richard,

Thanks for writing (and for other solutions you've provided in the past).

I ran a packet tracer on the FTD 1150 and it allowed the traffic from a vlan interface IP on the core to my google target IP so I took that as confirmation that natting is in place.

Is there a more appropriate test?

Thanks again,

Bob

[Richard Burts]

Bob

You are welcome. packet tracer is a good start and if it indicates that traffic is allowed that is a good indication. I would be inclined to examine the configuration and verify that it has logic to handle all of the subnets that might use it for access to Internet. A different test would be to use traceroute (or tracert depending on OS of the device) rather than ping. If the traceroute shows alternating responses between 2 IP addresses this would verify that you have a loop.

[Georg Pauwen]

Hello,

if possible, post the full running configs of both core switches and both firewalls...

[BobGreer65666]

Hi Georg,

The exec FW is a Sophos XG310 and doesn't have an equivalent to 'show run' (damitol).

[Giuseppe Larosa]

Hello @BobGreer65666 ,

examining the configuration of your core switches and of FTD I have seen the following:

The two cores are doing routing using VLAN 255 OSPF process 1  area 0

On first core switch EX-CORE-01 we see:

ip route 0.0.0.0 0.0.0.0 10.0.0.251 name Local_Egress_Sophos track 1
ip route 10.0.55.31 255.255.255.255 192.168.20.31 track 31
ip route 0.0.0.0 0.0.0.0 10.1.0.1 250 name NWI_Core

track 1 ip sla 1 reachability
delay down 60 up 15
!

ip sla 1
icmp-echo 8.8.8.8 source-ip 10.0.0.2
frequency 5

where IP SLA 1 is in vlan 10

interface Vlan10
ip address 10.0.0.2 255.255.255.0
no ip redirects
ip ospf dead-interval 9
ip ospf hello-interval 3
ip ospf priority 255
ip ospf 1 area 0
!

I have some questions for you:

1) your backup floating default static route points to a next-hop that should be on NWI_Core according to name

>> ip route 0.0.0.0 0.0.0.0 10.1.0.1 250 name NWI_Core

However, looking at NWI_Core configuration NWI-CORE-01 is the SVI VLAN 11 IP address

2) on first switch  EX-CORE-01 you are also running EIGRP AS 110 and you havefew  static routes like the following:

ip route 10.0.15.0 255.255.255.0 10.0.0.252 name DMV_V115

and many static routes like the following:

ip route 192.168.82.0 255.255.255.0 10.0.0.250 name AGAVE

what are the devices with those IP addresses 10.0.0.252 and 10.0.0.250 ?

your primary default route points to 10.0.0.251 that I suppose is the firewall to the internet.

to be noted on NWI core there is a single default static route

>> ip route 0.0.0.0 0.0.0.0 10.0.0.2

the FTD NWI has the followiing inside

ip address 10.1.0.252 255.255.255.0

who is 10.0.0.2 ?

Hope to help

Giuseppe