Linux server routing / CSS load balancing

[emilyharris]

I'm using the CSS Basic configuration guide as a starting point for a basic load balancing setup with two web servers. The two web servers are running Linux Fedora Core, and each have multiple network interfaces. One interface on each linux box is "public" for admin purpose only, one is "private" and connected to the CSS.

The basic CSS configuration works great...ONLY if the public IP address on the Linux box is totally disabled. I'm currently in heated debate with our Linux sysadmin (I'm the router guru, he's the Linux guru) as to what the issue is. It could be a routing issue on the Linux (my vote) or some issue on the CSS (his vote).

With the public interface enabled, I can ping the VIP (makes sense, since the CSS replies), and "show summary" shows the HTTP connections (and others, such as SSH) occurring round-robin as they should. However, on my client the connections time out. And again, when the public interface is disabled, it works fine.

Really, what I need is the CSS configuration guide that explains how the CSS returns info to a client and/or web server. My hunch is that the web servers are trying to return information to a public IP, and therefore going out the wrong interface (the public one), and not via the CSS. But the other argument from the sysadmin is that the CSS "from" IPs should all be private, and therefore no routing is involved on the Linux side; it should be returning its HTTP information to the CSS and that is that.

I hope that made some sort of sense. Anything in the right direction would be very helpful!

[Gilles Dufour]

You're linux admin is not really a guru ;-)

By sniffing the connection on the server [tcpdump], he could have seen that the CSS reuses the client ip address when sending the traffic to the server.

Therefore, if the server has 2 interfaces, it may respond to the client ip directly, bypassing the CSS which would be a problem.

If you don't need to see the client ip on the server, you can tell the CSS to use a VIP address to connect to the server.

The command is

group client-nat

vip x.x.x.x

add destination service

active

!

Regards,

Gilles.

[emilyharris]

We *do* need the client address. That's the current snafu. It is my opinion that the Linux server needs to be told "packets from eth1 go back out eth1" and "packets from eth0 go back out eth0". But, our Linux guy seems to think Cisco can use a form of IP source-routing to encapsulate routing information in the packet, making the Linux server's opinions obsolete. ie, bundle the route (back through the incoming interface) into the packet going to the Linux server. I personally have only used source-routing in the realm of token-ring/ethernet bridging, so I am not entirely sure how it works in Cisco IOS, REAGARDLESS of the fact that the CSS isn't truly IOS, and doesn't have all the features of classic Cisco routers/switches.

Is there anyone else with a similar situation who doesn't use NAT to solve this?

[Gilles Dufour]

I don't think Linux can route back to the interface traffic came from.

Linux has a routing table like any router and if the default gateway is pointing out eo, traffic will be sent back to e0 even if the request came in on e1.

You have to fix the routing table of your linux device.

If your linux *expert* is saying the CSS is doing fancy stuff, just tell him the css is a nating device.

All the CSS does is nat the traffic from VIP address to server address.

We don't have source routing to fix this problem.

You either nat or change your routing table.

Gilles.

[jfoerster]

HI,

from my point of view there are 3 possible solutions:

1) fix the routing table on the linux box. Btw your admin should know that there is a routing problem by having a look at a snoop or tcp-dump as Gilles mentioned it and on the other hand by just thinking that there is a problem when you need the original IP-address at the Web-Server...

2) do policy routing on the router/default gateway behind the linux box to forward the return traffic the linux box sends out towards the CSS (very bad solution and hard to troubleshoot)

3) Check if there is a possibility to add the original IP-Address in the HTTP header like it is done by proxy with the x-forwarded-for or by the CSM with the IP-Header insert functionality (see http://www.cisco.com/en/US/customer/products/hw/modules/ps2706/prod_bulletin09186a00802072b0.html)

This allows to log the original IP-addresses by the web-server if he checks for this attribute and allows to do "Source-Nat".

Unfortunately I haven't found such a feature on the CSS so far but maybe I did not dig deep enough in the config guides.

Kind Regards,

Joerg Foerster