topic Re: How do I load balance TFTP between two servers and a client in Application Networking

How do I load balance TFTP between two servers and a client on the same subnet?

pjwhitby_2 — Tue, 19 Apr 2011 03:19:15 GMT

Hi,

I have trawled through several documents and tried umpteen different configs, all to no avail. I have a PXE boot client trying to access a boot file via TFTP from a couple of TFTP servers on the same VLAN/subnet. For HA purposes I want to load balance the two TFTP servers.

Config is currently;

=====

probe icmp ICMP_PROBE

description icmp probe for default gateway tracking

interval 5

passdetect interval 15

rserver host server1

description Server1

ip address 10.0.0.1

inservice

rserver host server2

description Server 2

ip address 10.0.0.2

inservice

serverfarm host serverfarm_01

description servers used

probe ICMP_PROBE

rserver server1

inservice

rserver server2

inservice

class-map match-all L4_VIP_TFTP

10 match virtual-address 10.0.0.10 udp eq 69

policy-map type loadbalance first-match L7_TFTP

class class-default

serverfarm serverfarm_01

policy-map multi-match L4_LB_VIP_POLICY

class L4_VIP_TFTP

loadbalance vip inservice

loadbalance policy L7_TFTP

loadbalance vip icmp-reply active

nat dynamic 1 vlan 200

interface vlan 200

ip address 10.0.0.250 255.255.255.0

nat-pool 1 10.0.0.241 10.0.0.243 netmask 255.255.255.255 pat

service-policy input L4_LB_VIP_POLICY

no shutdown

ip route 0.0.0.0 0.0.0.0 10.0.0.254

=====

I have read the doco by Ivan Kovacevic amongst many others but as my clients and servers are on the same subnet, the config doesnt work.

Can anybody point me in the right direction please. The devices are ACE 4710 running A3(2.3).

Thanks

Re: How do I load balance TFTP between two servers and a client

pablo.nxh — Tue, 19 Apr 2011 03:40:17 GMT

Hello there,

Have you tried using the same IP (10.0.0.10) for NATing the response back from your TFTP server?

Regardless of the one-armed mode you still need the dummy (all zero) virtual to match the UDP random port sent from your servers.

If you haven't tried with that kind of NAT let me know and I'll drop you a config sample

Tnx

__ __

Pablo

Re: How do I load balance TFTP between two servers and a client

pjwhitby_2 — Tue, 19 Apr 2011 05:50:35 GMT

Hi Pablo,

I knew it would something on the NAT as I can see the connections in 'show connection'. If you have a sample config for this example that would be awesome.

Thanks alot,

Re: How do I load balance TFTP between two servers and a client

pablo.nxh — Mon, 25 Apr 2011 18:22:08 GMT

Hi,

Try simply changing the nat-pool or create a new one but using the same IP as the VIP. (The decision depends whether you have other protocols

being balanced besides TFTP),

class L4_VIP_TFTP

loadbalance vip inservice

loadbalance policy L7_TFTP

loadbalance vip icmp-reply active

nat dynamic 2 vlan 200

interface vlan 200

ip address 10.0.0.250 255.255.255.0

nat-pool 1 10.0.0.241 10.0.0.243 netmask 255.255.255.255 pat

nat-pool 2 10.0.0.10 10.0.0.10 netmask 255.255.255.255

service-policy input L4_LB_VIP_POLICY

no shutdown

HTH

__ __

Pablo

Re: How do I load balance TFTP between two servers and a client

pjwhitby_2 — Wed, 27 Apr 2011 01:53:58 GMT

Hi pablo,

thanks for the config, but it didnt work, here is my current config,

=====

probe icmp ICMP_PROBE

description icmp probe for default gateway tracking

interval 5

passdetect interval 15

rserver host server1

description Server1

ip address 10.0.0.1

inservice

rserver host server2

description Server 2

ip address 10.0.0.2

inservice

serverfarm host serverfarm_01

description servers used

probe ICMP_PROBE

rserver server1

inservice

rserver server2

inservice

class-map match-all L4_VIP_TFTP

10 match virtual-address 10.0.0.10 udp eq 69

policy-map type loadbalance first-match L7_TFTP

class class-default

serverfarm serverfarm_01

policy-map multi-match L4_LB_VIP_POLICY

class L4_VIP_TFTP

loadbalance vip inservice

loadbalance policy L7_TFTP

loadbalance vip icmp-reply active

nat dynamic 2 vlan 200

interface vlan 200

ip address 10.0.0.250 255.255.255.0

nat-pool 1 10.0.0.241 10.0.0.243 netmask 255.255.255.255 pat

nat-pool 2 10.0.0.10 10.0.0.10 netmask 255.255.255.255 pat

service-policy input L4_LB_VIP_POLICY

no shutdown

ip route 0.0.0.0 0.0.0.0 10.0.0.254

=====

I have the following in the 'show conn detail' output

conn-id np dir proto vlan source destination state

----------+--+---+-----+----+---------------------+---------------------+------+

43455 1 in UDP 212 10.0.0.200:52007 10.0.0.10:69 --

[ idle time : 00:00:02, byte count : 79 ]

[ elapsed time: 00:00:02, packet count: 1 ]

1847713 1 out UDP 212 10.0.0.2:69 10.0.0.10:52007 --

[ conn in reuse pool : FALSE]

[ idle time : 00:00:02, byte count : 0 ]

[ elapsed time: 00:00:02, packet count: 0 ]

and then the connection times out after approx 3 minutes with no increase in the latter connections byte or packet count.

I am guesssing the packet or request contains the clients real IP address and the server is trying to talk directly to the client rather than the LB VIP?

Can anyone suggest the next step, I am running out of ideas?

Thanks

Re: How do I load balance TFTP between two servers and a client

pablo.nxh — Wed, 27 Apr 2011 15:48:03 GMT

Hi,

Oh I thought the dummy virtual was already in place, please configure the highlighted portion.

class-map match-any TFTP    
   match virtual-address 0.0.0.0 0.0.0.0 udp eq any

policy-map multi-match L4_LB_VIP_POLICY

class L4_VIP_TFTP

loadbalance vip inservice

loadbalance policy L7_TFTP

loadbalance vip icmp-reply active

nat dynamic 2 vlan 200

  class TFTP
   nat dynamic 2 vlan 200

* I don't see the ACL permitting traffic into VLAN 200, did you remove it from the example? 

HTH
__ __ 
Pablo

Re: How do I load balance TFTP between two servers and a client

pjwhitby_2 — Wed, 27 Apr 2011 23:55:31 GMT

Hi pablo,

thanks for sticking with this one its really appreciated, but it still isnt working. I did have the ip any any for access into the vlan, heres is the current config

Theres is now a second VIP 10.0.0.201 as I am running through this case with Cisco TAC and I wanted to keep both yourself on this thread and the Cisco TAC engineer active, if I get an answer from either method I will let everybody know.

and sorry, the VLAN is VLAN212 not VLAN200.

====

access-list ALL line 10 extended permit ip any any

probe icmp ICMP_PROBE

interval 5

passdetect interval 15

rserver host server1

description Server1

ip address 10.0.0.1

inservice

rserver host server2

description Server2

ip address 10.0.0.2

inservice

serverfarm host serverfarm_01

description servers used

probe ICMP_PROBE

rserver server1

inservice

rserver server2

inservice

class-map match-all L4_VIP_TFTP

10 match virtual-address 10.0.0.10 udp eq 69

class-map match-all L4_VIP_TFTP2

10 match virtual-address 10.0.0.201 any

class-map match-any TFTP

10 match virtual-address 0.0.0.0 0.0.0.0 udp any

policy-map type loadbalance first-match L7_TFTP

class class-default

serverfarm serverfarm_01

policy-map type loadbalance first-match L7_TFTP2

class class-default

serverfarm serverfarm_01

policy-map multi-match L4_LB_VIP_POLICY

class L4_VIP_TFTP

loadbalance vip inservice

loadbalance policy L7_TFTP

loadbalance vip icmp-reply active

nat dynamic 2 vlan 212

class L4_VIP_TFTP2

loadbalance vip inservice

loadbalance policy L7_TFTP2

loadbalance vip icmp-reply active

nat dynamic 3 vlan 212

class TFTP

nat dynamic 2 vlan 212

interface vlan 212

ip address 10.0.0.250 255.255.255.0

access-group input ALL

nat-pool 1 10.0.0.241 10.0.0.243 netmask 255.255.255.255 pat

nat-pool 2 10.0.0.10 10.0.0.10 netmask 255.255.255.255

nat-pool 3 10.0.0.201 10.0.0.201 netmask 255.255.255.255

service-policy input L4_LB_VIP_POLICY

no shutdown

ip route 0.0.0.0 0.0.0.0 10.0.0.254

====

Please feel free to ask any questions, I am on BNE time.

Thanks

Paul

Re: How do I load balance TFTP between two servers and a client

pjwhitby_2 — Tue, 03 May 2011 00:55:39 GMT

Hi pablo,

Do you have any thoughts or configs I can try please.

If anybody else has any ideas, please feel free to jump in, just to remind everybody.

I have two TFTP servers that will be issuing a boot file (ardbp32.bin) to PXE booting clients. The two TFTP servers reside within the same VLAN as the PXE booting clients and in order to supply a degree of HA, I want to place the two TFTP servers (.1 and .2) behind a VIP on a ACE4710 (.10). When we distribute the TFTP server address via DHCP (option 66) I want to send the VIP address (.10) and allow the ACE4710 to load balance the TFTP conversation.

Thanks to all,

Paul

Re: How do I load balance TFTP between two servers and a client

udid — Sat, 07 May 2011 18:26:09 GMT

Try using the following configuration:

Note: Please make sure to configure also a udp probe to probe udp port 69, in case the application is down.

You need to configure a management policy on the interface when using a UDP probe.

That is because, when port 69 on the server will be unreachable, the server will send an ICMP unreachable.

ACE will consider a udp probe as "failed" only when it sees ICMP unreachable.

Without a management policy-map, the ICMP unreachable message will be dropped.

Also, add an ICMP probe to the rserver because udp probe will not be enough when the physical interface will be down.

That is because UDP is a connection-less protocol. To consider a UDP probe successfull, ACE need to see NO answer from the server in respose to the probe.

The ACE will not see any answer from the server when the interface is down and thus, will consider the probe as "sucessful".

With ICMP probe attached to the rserver, you also test the reachability of the server and not only the UDP port.

Here is the configuration (of course, you can chage the names of the of the objects to the name you are using if you want) :

access-list ALL line 10 extended permit ip any any

probe udp TFTP

port 69

interval 5

passdetect interval 15

probe icmp ICMP_PROBE

interval 5

passdetect interval 15

rserver host TFTP_1

ip address 10.0.0.1

probe TFTP

probe ICMP_PROBE

inservice

rserver host TFTP_2

ip address 10.0.0.2

probe TFTP

probe ICMP_PROBE

inservice

serverfarm host TFTP-SFARM

rserver TFTP_1

inservice

rserver TFTP_2

inservice

sticky ip-netmask 255.255.255.255 address source TFTP-STICKY

timeout 10

replicate sticky

serverfarm TFTP-SFARM

class-map type management match-any MANAGE

2 match protocol icmp any

class-map match-all NAT

2 match virtual-address 0.0.0.0 0.0.0.0 udp any

class-map match-all TFTP

2 match virtual-address 10.0.0.10 udp eq 69

policy-map type management first-match MANAGE

class MANAGE

permit

policy-map type loadbalance first-match ROUTE

class class-default

forward

policy-map type loadbalance first-match TFTP-POL

class class-default

sticky-serverfarm TFTP-STICKY

policy-map multi-match TFTP-MULTI

class TFTP

loadbalance vip inservice

loadbalance policy TFTP-POL

nat dynamic 1 vlan 212

class NAT

loadbalance vip inservice

loadbalance policy ROUTE

nat dynamic 2 vlan 212

interface vlan 212

ip address 10.0.0.250 255.255.255.0

no normalization

access-group input ALL

nat-pool 1 10.0.0.241 10.0.0.243 netmask 255.255.255.0 pat

nat-pool 2 10.0.0.10 10.0.0.10 netmask 255.255.255.0 pat

service-policy input TFTP-MULTI

service-policy input MANAGE

no shutdown

Let me know how it goes.

Good luck!

Re: How do I load balance TFTP between two servers and a client

pjwhitby_2 — Mon, 06 Jun 2011 03:57:06 GMT

I'm back from leave so I can now reply,

thanks for your response Ehud but that didnt work either.

I have another test config from TAC that I am going to try and I will then re-visit your solution and try again but I wanted to re-post this (bump it) to see if anybody out these has successfully used a Cisco ACE to load balance TFTP?

Thanks

Paul

Re: How do I load balance TFTP between two servers and a client

pjwhitby_2 — Tue, 05 Jul 2011 07:29:07 GMT

sorry, I still don't have a solution but I wasn't able to work on the issue last week.

anybody any thoughts?

How do I load balance TFTP between two servers and a client on t

tonesco01 — Tue, 17 Apr 2012 15:00:36 GMT

Hi pjwhitby,

I'm having the same problem here.. So, I get the solution?

How do I load balance TFTP between two servers and a client on t

pjwhitby_2 — Tue, 17 Apr 2012 23:38:31 GMT

hey Wallace,

sorry, we never got this running and we worked in a work around, although I cannot remember what it was, I think we went with a round robin DNS solution, but I would have to check.

Paul

How do I load balance TFTP between two servers and a client on t

chrhiggi — Thu, 19 Apr 2012 20:28:59 GMT

All-

You can not loadbalance TFTP one-armed (servers and clients on the same vlan). Here is why-

The way tftp works, the client sends a UDP packet with a random source port destine to the server on port 69. The server responds with source port 69, however, instead of using the client port as a destination like most protocols, TFTP generates a random port number to send data traffic back to the client.

ACE tracks connections via MAC addresses, IP's and port numbers, and ingress/egress interfaces on a connection. When we send the packet out of the interface to the server, we are expecting a response back from to the clients port. To ACE, the port randomization the server is doing looks like a brand new UDP connection coming from the server.

As relates to loadbalancing TFTP - The client is talking to the VIP IP and expects to recieve a response from that IP. The server is talking to the client IP and is communicating with ACE. ACE has to catch the server's traffic destine to the client and modify the source IP from the server back to the VIP. When the server replies to the client connection with a new port, you can catch it with a class map that matches UDP source port 69 and source-nat the packet to the VIP IP. That is what was being done above via this configuration:

class-map match-any TFTP

10 match virtual-address 0.0.0.0 0.0.0.0 udp any

class NAT

nat dynamic 2 vlan 212

However - with one-armed configurations, the gateway on the server is not ACE. That means, the client sends a packet to the VIP, the ACE forwards it to the server IP, the server sends its response back to the client. At that point, the client is confused because it was talking to the VIP IP, but recieved a response from the Server IP. To mitigate that, you need to configure source NAT on the connection going from ACE to the server. That exact part is what makes it impossible to loadbalance TFTP one-armed.

Think about this:

When you configure source nat, many clients use one single IP. Source ports are changed to track which client belongs to which connection on the backend. (since all clients are going to use the same single IP between the ace and the server, how else are you going to track the connection?)

So, when the server is going to reply to the NAT pool IP and the destination port is going to be random - How exactly would the ACE know what client in the NAT table is supposed to recieve that packet? It can't.

The only possible way to get this to work is configure policy based routing on the gateway to force any source port 69 traffic from the server back to the ACE so that ace can Un-Nat the traffic with a standard tftp loadbalancing configuration. In that manner, you are replacing the function of source nat by routing. You just need to ensure that the packet goes into the interface that ACE egressed the first udp packet to the server on.

I know that this is complicated to understand by reading, if you would like a set of wireshark traces to suppliment this, email me: chrhiggi@cisco.com. I have both a standard TFTP connection as well as a single trace showing both sides of TFTP being loadbalanced.

Regards,

Chris Higgins