Static routing on a switch to a specific server

[Damon M.]

Hi guys,

In our environment we work with a router on stick kind of setup combined with HA.

Now, recently we added a new switch (Catalyst 9500) in our network with 40Gbps uplink and 10Gbps connections straight to some of our clients.

Now this is partially our setup and the routing that i'd like to talk about :

Our firewall goes to multiple switches, either for our clients, to our servers, and others. Clients have their own set of switches and so do our servers.

Now, when you look at the picture you can see 3 clients to a switch. In my case that would be our new Catalyst 9500. That goes to a LAN core switch and then to the firewall (which also serves as router btw).

Now that all works fine. What we would like to do is the following :

When you look at the bottom right server called file. It is both connected to our server switch and to the 9500 catalyst switch.

The server switch is connected to a regular 1Gbps connection on the server, while the catalyst is connected to a 40Gbps ethernet connection.

Normally, all of our user (even the ones that are not on this picture) go to the Firewall/Router, then to the server switch and so access the file server.

What we would like for those 3 clients, is that they do NOT go to the firewall/router but instead take a shortcut straight to the file server over 40Gbps and this is where i am stuck.

For the clients we work with VLAN's (lets say VLAN 45 for the clients) and the servers just work on the default vlan 1. There are firewall rules in place to keep them separated and safe.

What i did so far is i gave ForyGigabitEthernet 1/1/2 a static IP and i used that IP as the gateway on the fileserver and that connection works. I can ping the server from the 9500 and i can ping the interface on the switch from the server.

What i cannot is ping the server from the clients. I made an static ip route towards the server but i dont know how to get the clients towards the server instead of going towards the firewall/router.

Here is the config i can give :

File server config :

IP : 10.32.1.98 255.255.255.0

Default gateway : 10.32.1.97

interface FortyGigabitEthernet1/1/2
description CONNECTIE FILESERVER -> 10.32.1.98
no switchport
ip address 10.32.1.97 255.255.255.0

ip route 10.32.1.98 255.255.255.255 FortyGigabitEthernet1/1/2 (in the hope to force trafic towards that ip over this port).

---

My main question, how do i get this done without screwing both networks, the default gateway for all of the clients is the firewall/router : 10.32.45.2

I hope i gave all the information needed to solve this puzzle ^^ i feel like i'm missing a basic piece of networking but i'm looking over it.

If you guys could help that would be awesome!

Greetings & thanks in advance,

Damon

[Elliot Dierksen]

If you are crossing L3 networks (like between VLAN's), there is no way to do that. It has to go through the device that routes between the different L3 networks. If they are on the same L3 network, the switches should send the traffic to that server via the optimal path for the MAC address of the server based on spanning tree. This sounds like a job for an L3 switch. IMHO, firewalls make poor L3 core switches.

[Damon M.]

I've updated my example hoping it makes a bit more clear of whats going on.

Firewall in question is not a Cisco firewall/router but a PFSense solution.

That being said, the Catalyst 9500 is a L3 switch.

[paul driver]

Hello

---

@Damon M. wrote:

ip route 10.32.1.98 255.255.255.255 FortyGigabitEthernet1/1/2 (in the hope to force trafic towards that ip over this port).

That being said, the Catalyst 9500 is a L3 switch.

---

The static route isnt applicable

If the L3 vlan for 45 is residing on the catalyst 9k then you can append a policy based route to so traffic from that subnet to the file sever will be routed via the 40GB interface,

Catalyst 9K
access-list 100 permit ip any host 10.31.1.98

route-map PBR-Fileserver
match ip address 100
set ip next hop 10.31.1.98

int vlan 45
ip policy route-map PBR-Fileserver

[Damon M.]

I have done what you said but unfortunately i am stil not able to ping 10.32.1.98 from the clients side. Although i do feel we're going the right direction.

[Elliot Dierksen]

Regardless of what it is, firewalls still make poor L3 cores. IMHO, you would be much better off making the Cat 9500 the core of your network. My suggestion would be to have only one L3 interface on the firewall. Assign all the other IP's to SVI's (switched virtuual interface, aka a VLAN interface) on the 9500. I would even keep a separate network between the 9500 and the firewall that is only used for that. You could do that if you have the 9500 take over all the IP addresses that are currently owned by the firewall. The wrinkle in this is if there were some things you wanted protected by the firewall. If there are no restrictions on traffic between hosts inside the firewall, then using the 9500 as the L3 core is definitely the way to go.

[Damon M.]

Thanks for the info, going to try this out and get back to you.

[paul driver]

Hello

i am stil not able to ping 10.32.1.98 from the clients side

At this time it hard to understand what the issue is but if your mixing the L3 routing between the the rtr and the switch then that could be a problem 

The PBR should work if as i said the L3 vlan for those clients reside on the switch if it doesn’t then you may need to add to additional static routing or manipulate the igp routing you may be running

Can you post the config of the L3 catalyst and the rtr into a file and attach also add

sh ip protocols 

sh ip route 

[Damon M.]

Hey Paul,

I think some stuff may be losed in translation because we're not entirely sure with what you with "reside on the switch"

Also, i'm not really sure what you mean with rtr?

I'll give you the show's as you asked :

sh ip protocols

LAN_DISTRO_MARKETING#sh ip protocols
*** IP Routing is NSF aware ***

sh ip route 

LAN_DISTRO_MARKETING#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.32.1.0/24 is directly connected, FortyGigabitEthernet1/1/2
L 10.32.1.97/32 is directly connected, FortyGigabitEthernet1/1/2

[Elliot Dierksen]

One other thing that is a key point is that L2 enters into this as well. You may have an IP route to a specific device, but the actual physical path taken will be based on the MAC address of the target device. Is there a physical connection between the server core and the lan core switch? Also it appears that the 10.32.1.98 file server has physical connections to both the server core and the cat9500. Is it only using a single IP address? If so, then it is probably bridging between its two connections which could be causing the L2 traffic to not be taking the path you expect. I am an infrastructure guy, so I would never want a server acting as a bridge unless there was an unusual use case that required that.

[Damon M.]

Hey Elliot, thanks a lot for your input! We're aware that we dont have the best use case running and are taking all of your points with us for future changes and upgrades once we get there. Currently we have a running solution for the entire company (except for this on issue than) which has proved its reliability so far. Once we get passed our current priority projects we might take a look at our current network and go for improvement such as the core setup that you pointed out earlier. We're aware but not able to do such things in this point in time but do appreciate it!

Now to get to your second point. 

1) No there is no physical connection between the Server core and Lan core, these are kept separate. 

2) There are 2 connections on the file server

1. Onboard motherboard connection to the server core (1Gbps and a different IP as well , 10.32.1.96)
2. 40Gbps PCIe slot card going to the C9500 configured with the above mentioned IP's (10.32.1.98)

All clients in our network take their regular route towards the file server which is " Client -> Distro -> Lan Core -> Router/Firewall -> Server core -> File server (10.32.1.96)

What we're trying for our marketing dep. is going from & to " Client -> Catalyst 9500 -> File server (10.32.1.98) " 

The reason the C9500 also has a connection to the Lan Core is so that all other traffic can still follow the route they're meant and used to take. We only want to shortcut traffic towards the file server for that 1 department because of the amount of content they use.

I hope this clarifies everything a bit more. Looking forward to hear more!

[Elliot Dierksen]

I think you are looking at this as a layer 3 when I think the path selection issues are really a layer 2 thing. Looking at your diagram, the firewall is the only thing that has an interface in both the server and client VLAN's. I assume it is the default gateway for the client machines which would make sense. That means to get to that server via the 40G interface in the 9K, the packet has to go from the 9K to LAN core to the firewall. This is because the client ARP's for the firewall's IP, and then sends a layer 2 ethernet packet to the gateway. The firewall then routes the packet from client VLAN interface to server VLAN interface. Then it ARP's for the server IP, and then sends a layer 2 ethernet packet to the LAN core to the 9K, then to the server. The 9K has no IP connectivity to the server VLAN, so it has to hairpin through the firewall. That means your 40G interface isn't getting you much. This is even true when something on the server core tries to talk to the 40G IP of that server. The server would ARP for the 40G IP. Then the server would send a layer 2 ethernet packet to the server core which would send it to the firewall on one LAN port, then back out the LAN port of the firewall, then to the LAN core, then the 9K and then to the 9K, and on to the 40G interface. I am assuming from the drawing that there isn't a direct connection between the LAN core and the server core. If that is true, even packets on the same VLAN have to traverse the firewall to get to the 40G IP. That would likely give you less throughput than going to the 1G IP. Your topology is going to limit the usefulness of that 40G interface.

[Damon M.]

I am aware that this topology is a limiting factor because indeed, the firewall is the gateway for all of the clients and server. I was just hoping if there would be some way of short cutting it so that when ever a client would search for 10.32.1.98 that the switch would force it towards that interface instead of going towards the firewall/gateway but by now i presume there is no way because of its current design. am i right?

And if so, may i ask you why you think that the solution provided by Paul does not work?

[Jon Marshall]

Paul's solution will not work because as I understand it it would require the default gateway of the clients to be on the 9500 switch and then you could use PBR, but it isn't, it is on the pfSense. 

What you need is to put marketing and the 40Gbps NIC on the server into a new vlan and route that on the 9500 but it is not clear how easy that would be to implement in terms of connectivity to everything else without knowing more about the links. 

As Elliot has described this is really a L2 issue you have and you need it be a L3 one. 

Jon

[Richard Burts]

I have looked through the config that was posted. While there is much that I have not looked at closely I do see several problems"

- the biggest problem is that you are applying the ip policy route-map on a vlan interface that has no IP address. If IP is not processing on that vlan interface then PBR will not work.

- a secondary issue, if I am understanding the objective correctly, is that the access list permits any source. I thought that the objective was that only 3 specified client machines would use the new 40 G path. If you want only 3 sources to use this path then the acl needs to specify those sources.

- If the PBR is corrected it should allow the 3 clients to use the alternate path to get to the server. But how will the server get back to those 3 clients? It would appear that the server would send its responses back on the normal path to the firewall. And then the firewall would see response traffic for which it has not seen a request. And I am guessing that the firewall would deny that.