VLan/Routing issues

[Ron M.]

Hi, I'm fairly novice in terms of routing and network.

But nonetheless, I've inherited a network that simply doesnt work all the time.

From what I can tell... I have three network devices, Cisco3750, 3650 and a Fortigate 100D. The design of the network is to support policy based firewalling between three networks 169.210.0.0/16, 169.212.0.0/16 and 169.213.0.0/16 via the Fortigate.

There are a number of servers connected to the 3750 and 3650 for each network 169.212.0.0/16 and 169.213.0.0/16 are in Vlan 212/213 respective. Servers in the 169.210.0.0/16 use the 169.210.20.3 (3750 router) as their default route, the 169.212.0.0/16 use 169.212.40.1 on the Fortigate and the 169.213.0.0/16 use 169.213.40.1 on the Fortigate.

There are no IP address assigned to either Vlan 212 or Vlan 213 on the Cisco switches, but both switches have the vlans in their databases. The Fortigate is hanging off two dot1q trunks one trunk allowing Vlan 212 and the other allowing Vlan 213. On the Fortigate iindivudal ports (no ip address on the ports) with sub interfaces defined in Vlan 212 and Vlan 213. Each sub interface has an IP address 169.212.40.1 and 169.213.40.1 and it is used by the devices in those networks as their default gateway

From time to time, devices on the 169.212.0.0/16 or the 169.213.0.0/16 cannot ping devices on 169.210.0.0/16, most of the time they can connect just fine. The ping traffic simply stops and eventually starts back up. Sometimes it takes restarting networks (linux) or restarting computers (windows). This keeps things working for hours to days...but eventually traffic stops moving.

I can see (wireshark) a ping request from a 169.213.0.0/16 machine go out, then see the request on the 169.210.0.0/16 machine, but I dont see the response when things are not working. The MAC address of the pings suggest that the pings are taking the default gateway route and there is no route to the 169.212.0.0/169.213.0.0 network.

When things are working, the pings look good request/response; however the MAC address of the default route is being used, so its the same source/dest working or not.

Show ip route does not show the static route being installed for either the 169.212.0.0/16 or the 169.213.0.0/16 network From the 3750, traceroute to 169.212.40.48 says it is using the default gateway 169.210.10.2 and not the 169.212.40.1 as described in the static routes.

How can I get traffic to move from the 169.212.0.0/16 and 169.213.0.0/16 reliably to the 169.210.0.0/16 network?

Secondly, are there any really good tools to help show traffic flows besides wireshark?

attached is a diagram of the networks/devices

thanks ron

[paul driver]

Hello

try creating L3 svi for tthese vlans  on tthe 3750 

int vlan 212

ip address 169.222.0 1 255.255.0.0

int vlan 213

ip address 169.223.0.1 255.255.0.0

int vlan 210

IP address 169.210.0.1 255.255 0.0

Have you devices on these vlans use their default gateways as the 3750 svi interfaces and let this switch perform inter-vlan routing

Remove the statics and the IP default-gateway leave the default route

res

Paul

[Ron M.]

is 169.222 and 169.223 a typo?  The networks are 169.212 and 169.213.

ron

[paul driver]

Hello

Apologies Rolf not got my glasses on!

res

Paul

[Ron M.]

thats funny, I do that alot myself...thanks

[Ron M.]

Thanks Paul,

I'll give it a try, but wont this configuration bypass the Fortigate if its interfaces are not the default route for the devices?  

How can all the 212 and 213 network traffic be forced to go through the Fortigate interfaces so it can manage the connection policy (this can talk to that over this port...)?

ron

[Jon Marshall]

Yes it will bypass the firewalls.

So if the firewall setup is legacy and you don't need to firewall the vlans from each other Paul's suggestion is a good one.

If you need to firewall between them then you shouldn't do it.

You do have asymmetric traffic paths in your setup but I'm not sure that is causing the issue.

Jon

[Ron M.]

Yes, I need the firewall to control traffic flows via the Fortigate policies.

I thought I tried a configuration similar to his suggestion and saw that the Fortigate was bypassed.

How does one find/resolve 'asymetric traffic paths'...(I think I get the concept, but I need to google it to figure out what you are talking about).

I think I understand that the root cause is that the 3750 dosnt know how to get to 169.212.40.1 or 169.213.40.1 (show ip route - neither network shows up); but what I dont understand are all the possible reasons/remedies.

If the trunk ports to the Fortigate were connected directly to the 3750 and not have to travel through the 3650 switch would that simplify the paths?

ron

[Jon Marshall]

I am assuming for the non firewalled servers their default gateway is the 3750 ?

The pings from the switch will always use the default route.

The switch can't use the statics for the specific subnets because it has no interfaces in the same IP subnets as the next hop IPs so they are never used.

You would have to set the next hop IP to be the same as the default route next hop but there is no point because you have a default route.

The asymmetric path comes in because the firewall has an interface in the same IP subnet as the non firewalled servers so assume one of those servers pings a server in vlan 212 -

server ->  3750 -> firewall ->server (vlan 212)

but the return path is -

server (vlan 212) -> firewall -> server

notice that for the return traffic it does not go via the 3750 because the firewall has an interface in that IP subnet so it simply sends the traffic direct.

Again that is assuming the non firewalled servers have their default gateway set to the SVI on the 3750.

The way to fix it is to use a different IP subnet to connect the 3750 to the firewall so that traffic to and from the non firewalled servers has to follow the same path both ways.

However I am not saying this is your problem because it may well not be.

Jon

[paul driver]

Hello

Okay given that topology - All the routing between these vlans should go via the FW and I dont think you even need ip routing enabled on the 3750 or the two statics for vlan 212,213, just s DG point to pointing to the ip of the fortigate Port 1, unelss you need to point egress traffci to the wan?

So I think something is a miss?

But please confirm:
3750 -
i dont see it the L3 interface of vlan 210 - is this a SVI on this switch?
Does this have a trunk connected to the 3650

3650 -

3 trunks
1)   Connecting 3750
2/3) Connecting to the fortigate two subinterfaces?
no ip routing
no default-gateway

Servers
default-gateways of their relative L3 vlan interface


Wan
How are you getting out on the wan?

res
Paul

[Ron M.]

Hi,

169.210.0.0/16 is actually VLAN 1 (lan) on the 3750

3750

no ip source-routing

ip routing

ip default-gateway 169.210.10.2 << lan interface on Fortigate

ip classless

interface Vlan 1

ip address 169.210.20.3 255.255.0.0 

3650

Trunk from 3750 to 3650 all vlan

Trunk from 3650 to Fortigate allow vlan 212

Trunk from 3650 to Fortigate allow vlan 213

no ip routing

interface Vlan 1

ip address 169.210.20.5 255.255.0.0

ip default-gateway 169.210.20.3  <<< this may be a issue but that's that way it was

ip classless

Servers

all 169.210.0.0/16 - Lan have a default gateway of 169.210.20.3 (3750)

all 169.212.0.0/16 - VLAN212 have a default gateway of 169.212.40.1 (fortigate)

all 169.213.0.0/16 - VLAN213 have a default gateway of 169.213.40.1 (fortigate)

The wan comes in on another 3650 as VLAN 1000 (there are a set of 2 ports in vlan1000 one to the telco device and one to the Fortigate wan interface), so all wan traffic should go through the Fortigate.  There is another two interface ether-channel/trunk (all vlan) that goes back to the 3750.

The 3650 with VLAN 1000 looks like this

interface VLAN 1

ip address 169.210.20.4 255.255.0.0

ip default-gateway 169.210.20.3 << back to the 3750

ip classeless

The Fortigate has a static route

0.0.0.0 0.0.0.0 back to the telco gateway

I kinda see the reason not to route anything, as the Fortigate should take care of traffice flows.  So, given the answers to your questions...is that still a good idea to kill the routing alltogether?

Thanks for the advice so far, starting to make more sense; not all there yet, but I'm starting to understand the issue.

ron

[Jon Marshall]

As far as I can tell it is not a problem with any of your route statements.

Note that you keep referring to the "ip default-gateway .." command on the 3750 but that isn't used if the switch is routing.

What is used is the default route and you have that setup so your 3750 knows how to route to the firewalled vlans.

In terms of whether to route everthing on the firewall, that is really up to you.

There is no reason why you cannot route some vlans on the firewall and others on the switch and it should work.

It depends on what you are trying to achieve.

Jon

[paul driver · Accepted solution]

Hello

Please attached testing - 

The differance(s) is that I have

- all 210 servers are pointing to the fortigate ip for their DG
-ip routing disabled on the 3750

res
Paul

[Ron M.]

Yep, turning off the Cisco routing and pointing all the 169.210 servers to the Fortigate as their default gateway is the right way to fix the issue.

Been a few days now and everything seems to have settled in and working fine.

thanks for all the advice

ron