HI,
have a look at the routing table of the servers.
Is the return traffic (towards the clients) forwarded towards the loadbalancer from the servers or bypassing=
Are you using source limitation on the loadbalancer?
Are you using source nat?
Please paste the config of the loadbalancer, the routing table of the servers and the source-address that gives you a hard time and we can have a look at it.
Kind Regards,
Joerg
PS
IN case of any doubts take a sniffer trace in front of the loadbalancer and behind the load balancer. If necessary additional ones at the client and at the server