ACE in Direct Server Return mode not working as expected

Sebastian Helmer · ‎05-08-2013

Dear all,

I configured my ACE as I found it here:

https://supportforums.cisco.com/docs/DOC-22555

the VIP is working, that means I can ping it, routing is working etc.

I created a loopback on the win2012 Server with the IP of the VIP. When I try now to test the LB with telnet on port 25 e.g. it is not working. direclty on the server it works, also in my last deployment where I use SNAT/PAT. But we want the real client IPs visible on the Exchange Server.

Where is my problem ? Any ideas would be great..

rserver host YY

description AServer-1

ip address 10.1.x.2

inservice

rserver host XX

description AServer-2

ip address 10.1.x.3

inservice

serverfarm host Mail

description Mail

transparent

predictor leastconns

rserver AServer-1

inservice

rserver AServer-2

sticky ip-netmask 255.255.255.255 address both Mail

timeout 5

replicate sticky

serverfarm Mail

class-map match-all Exchange_ALL

2 match virtual-address 192.168.1.1 any

class-map type management match-any remote_access

2 match protocol xml-https source-address 10.a.b.0 255.255.255.0

3 match protocol icmp source-address 10.a.b.0 255.255.255.0

5 match protocol ssh source-address 10.a.b.0 255.255.255.0

7 match protocol https source-address 10.a.b.0 255.255.255.0

8 match protocol snmp source-address 10.a.b.0 255.255.255.0

9 match protocol xml-https source-address 10.d.e.1 255.255.255.255

10 match protocol icmp source-address 10.d.e.1 255.255.255.255

11 match protocol ssh source-address 10.d.e.1 255.255.255.255

12 match protocol https source-address 10.d.e.1 255.255.255.255

13 match protocol snmp source-address 10.d.e.1 255.255.255.255

policy-map type management first-match remote_mgmt_allow_policy

class remote_access

permit

policy-map type loadbalance first-match mail

class class-default

sticky-serverfarm Mail

policy-map multi-match VLAN20

class Exchange_ALL

loadbalance vip inservice

loadbalance policy mail

loadbalance vip icmp-reply

interface vlan 2

ip address 10.a.b.2 255.255.255.0

access-group input ALL

service-policy input remote_mgmt_allow_policy

no shutdown

interface vlan 20

description Server

ip address 10.1.x.20 255.255.255.0

peer ip address 10.1.x.30 255.255.255.0

no normalization

access-group input ALL

service-policy input VLAN20

no shutdown

ft interface vlan 4

ip address 10.f.g.2 255.255.255.252

peer ip address 10.f.g.1 255.255.255.252

no shutdown

ft peer 1

heartbeat interval 300

heartbeat count 10

ft-interface vlan 4

ft group 1

peer 1

associate-context Admin

inservice

ip route 10.d.e.0 255.255.255.255 10.1.x.1

ip route 0.0.0.0 0.0.0.0 10.a.b.1

Bilal Nawaz · ‎05-08-2013

Hello Sebastian,

Is this in a 'one arm' config mode?

class-map match-all Exchange_ALL

2 match virtual-address 192.168.1.1 any

Is the 192.168.1.1 address correct as the VIP? If so how will the load-balancer proxy arp for this address when it does not have an interface within this address range?

Should it not be an address in VLAN 20?

Hope this helps.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

Sebastian Helmer · ‎05-08-2013

This is just what I saw on the link I posted..

I'm not sure how that works in deep. But I can say that the VIP is alive and accessable from the client side by ping..e.g.

Just the, let me say, service is not working..

If u have any suggestions for a config, just let me know..I need to make sure, that the client IP is seen by the Server and not any NAT IP.

Bilal Nawaz · ‎05-08-2013

Hmm. I can't see anything obvious and you seemed to have followed the document correctly. This setup is new to me. That document says the ARP should be turned off on the server side to prevent it replying to requests. Since this has an 'asymmetric' feel to it you will get to the VIP, which is correct and then the only thing thats changed in the packet is the destination mac address which goes directly to the rserver. The rserver should then be able to respond directly to the client without traversing the ACE. What happens when you try telnet and do a 'show conn' on the ACE. What do you see?

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

Sebastian Helmer · ‎05-08-2013

The connection will timed out so not established. When I do it in my old setup with NAT it works...established and u can use the SMTP command like helo to see the client IP.

124857 1 in TCP 20 10.254.5.87:62501 192.168.1,1:25 ESTAB

121912 1 out TCP 20 192.168.1.1:25 10.254.5.87:62501 ESTAB

all other connections are removed ad management access and FT..

I do a telnet 192.168.1.1 25

Message was edited by: Sebastian Helmer

Bilal Nawaz · ‎05-08-2013

Are you able to establish whether or not the rserver is responding with the loopback address? Because when the client sends to the VIP - its expecting the VIP to respond back, not the actual RServer.

In our case we are bluffing this on the rserver with the loopback interface. But just need to make sure that it is actually responding back with this source.

If not, this will break the connection.

The NAT will work fine as the client is seeing the response to its query from the VIP.

What happens when you run wireshark on your PC/Laptop - this should help in seeing if you are getting anything back.

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

Sebastian Helmer · ‎05-08-2013

as I see it, I got nothing back from the VIP or the rserver.

I'm connected via VPN, but there should be no deny...but in the LAN at the afternoon it was the same..also from the Server network I tried by a colleauge the telnet.

It's really strange..I see on the core no arp for the VIP but maybe thats ok..

Bilal Nawaz · ‎05-08-2013

Hello, with NAT it will work. It will work because the client and reserver respond back to the IP that the NAT is.

Without NAT we need to ensure that the reserver is responding back with the VIP address (loopback interface).

You should be able to see what is happening more if you ran wireshark on your laptop so you can see the telnet going out and to see if anything comes back. If the reserver responds with its own IP then it will break the connection and not establish it.
Because, your laptop is expecting the VIP to reply back, but instead, the reserver will be replying back. You will not get successful telnet but you will be able to see from wireshark if you are getting a response back for your initial telnet port 25 request.
Your VIP is working and is responding to ICMP. There is reachability there so I don't think this is the problem here.

Hope this helps

Sent from Cisco Technical Support iPhone App

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

Sebastian Helmer · ‎05-09-2013

All right, I understand what u are saying. Thanks!
Tomorrow back in the office, I will check it. Via VPN I see no reply from the server or VIP. Maybe there is also something wrong on the server so I will have to speak with the server guys too.

Sent from Cisco Technical Support iPhone App

Sebastian Helmer · ‎05-11-2013

Ok, it works now with this change on the Server

http://blog.loadbalancer.org/direct-server-return-on-windows-2008-using-loopback-adpter/

netsh interface ipv4 set interface "HLB-VIP" weakhostreceive=enabled

netsh interface ipv4 set interface "HLB-VIP" weakhostsend=enabled

netsh interface ipv4 set interface "Local Area Connection" weakhostreceive=enabled

BUT, from LAN it is not fully working, telnet works but it is just "dark", when i do it from remote locations where a Firewall is in between, it works and I can use the SMTP helo, etc..

In Wireshark I can see the session and when I'm correct it tells me: BAD CHECKSUM and ip checksum offload and after a while i got a reset..

Bilal Nawaz · ‎05-11-2013

Glad to hear its working... sounds like you made good progress! When you say

telnet works but it is just "dark"

What do you mean? What is not happening that is meant to happen? Is this with testing telnet?

With regards to the BAD CHECKSUM and ip checksum offload, wireshark articles suggest it might be a setting on one of the NICs - maybe server side.

http://wiki.wireshark.org/CaptureSetup/Offloading

http://wiki.wireshark.org/TCP_Checksum_Verification

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

Sebastian Helmer · ‎05-11-2013

Nice to hear from u again.

I mean from remote we have no problem i do this: telnet "VIP" 25, i got a prompt and with helo I can see my client IP is seen by the Server.

When I do that from the LAN from a routed interface..that means....client--Router--Server with the ACE in my setup, the same as u see the link in my first post. I can establish the telnet but I get no prompt or reply from the server, so I can not use the helo command e.g.

Bilal Nawaz · ‎05-11-2013

Oh, I see. Very interesting indeed!

Do you get the BAD CHECKSUM and IP CHECKSUM OFFLOAD on the remote sites?

It could be this that is the problem. I read this and it seems as though it causes disconnects just as you experience too.

or just disable - it worked for some here, but for others, they upgraded the drivers of the NIC:

http://www.techsupportforum.com/forums/f137/wireshark-question-tcp-checksum-offload-248812.html

1. Open Device manager (right click "Computer" and click "Manage")

2. Click on "Device Manager"

3. Expand "Network Adapters"

4. Right click your network adapter

5. click "properties"

6. click the tab named "Advanced"

7. Find "IP Checksum Offload" and click it

8. Put the value to the right to "Disabled"

9. Find "TCP Checksum offload (IPvX)

10. Set the value to the right to "Disabled"

The Wiki Wireshark article had this:

In Windows, go to Control Panel->Network and Internet Connections->Network Connections, right click the connection to change and choose 'Properties'. Press the 'Configure...' button, choose the 'Advanced' tab to see or modify the "Offload Transmit TCP Checksum" and "Offload Receive TCP Checksum" values.

It seems like a server side issue rather than Load Balancer problem.

Hope this helps

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

Sebastian Helmer · ‎05-11-2013

I didn't check Wireshark but the test telnet ist still not working...I also think that is a client or Server issue or somethingin between I'm not aware of.

.

Sebastian Helmer · ‎05-11-2013

PS: On the netstat @the Server i see as State: SYN_RECEIVED