ACE 4710 no real ip in header with referer

Vladimir Buyalsky · ‎11-11-2013

Hello everybody. I need help with ACE 4710. We use one-arm PAT config to round robin load balance scheme with ssl terminate and trying to insert x-real-ip in all http-headers. But some packets with referer inside arrive with ACE ip in header, no real ip of clients. Any ideas, please?

Thanks!

Config ACE:

interface gigabitEthernet 1/1

channel-group 1

no shutdown

interface gigabitEthernet 1/2

channel-group 1

no shutdown

interface gigabitEthernet 1/3

channel-group 1

no shutdown

interface gigabitEthernet 1/4

channel-group 1

no shutdown

interface port-channel 1

switchport trunk native vlan 1

switchport trunk allowed vlan 2-200

no shutdown

crypto chaingroup intercert

access-list ALL line 8 extended permit ip any any

access-list ALL line 9 extended permit icmp any any

probe icmp ICMP_PROBE

interval 10

passdetect count 4

receive 1

probe http http_probe

interval 5

passdetect interval 10

passdetect count 2

expect status 200 210

header User-Agent header-value "LoadBalance"

probe http http_probe_443

port 443

interval 5

passdetect interval 10

passdetect count 2

expect status 200 210

header User-Agent header-value "LoadBalance"

rserver host nginx-1

ip address 10.99.1.11

inservice

rserver host nginx-2

ip address 10.99.1.12

inservice

rserver host nginx-3

ip address 10.99.1.13

inservice

rserver host nginx-4

ip address 10.99.1.14

inservice

rserver host nginx-5
ip address 10.99.1.15
inservice

rserver host nginx-6

ip address 10.99.1.16

inservice

rserver host nginx-7

ip address 10.99.1.17

inservice

rserver host nginx-8

ip address 10.99.1.18

inservice

serverfarm host NGINX-FARM-443

probe ICMP_PROBE

probe http_probe_443

rserver nginx-5

inservice

serverfarm host NGINX-FARM-80

probe ICMP_PROBE

probe http_probe

rserver nginx-1

inservice

rserver nginx-2

inservice

parameter-map type http HTTP

case-insensitive

persistence-rebalance

header modify per-request

set header-maxparse-length 16384

length-exceed continue

parameter-map type http HTTPS

persistence-rebalance

header modify per-request

set header-maxparse-length 16384

length-exceed continue

sticky ip-netmask 255.255.255.255 address source STUCK

serverfarm NGINX-FARM-80

timeout 60

replicate sticky

sticky ip-netmask 255.255.255.255 address source STUCK443

serverfarm NGINX-FARM-443

timeout 60

replicate sticky

action-list type modify http SERVER

header rewrite response server header-value "nginx" replace "hamster"

action-list type modify http REAL-SSL

header insert request x-real-ip header-value "%is"

ssl-proxy service SSL-PROXY

key xxx.em

cert xxx.pem

class-map match-all NGINX-443-VIP

2 match virtual-address 10.99.1.4 tcp eq https

class-map match-all NGINX-80-VIP

2 match virtual-address 10.99.1.2 tcp eq www

class-map type management match-any remote_access

2 match protocol xml-https any

3 match protocol icmp any

4 match protocol telnet any

5 match protocol ssh any

6 match protocol http any

7 match protocol https any

8 match protocol snmp any

policy-map type management first-match remote_mgmt_allow_policy

class remote_access

permit

policy-map type loadbalance http first-match NGINX-443-POLICY

class class-default

sticky-serverfarm STUCK443

insert-http X-Real-IP header-value "%is"

policy-map type loadbalance http first-match NGINX-80-POLICY

class class-default

sticky-serverfarm STUCK

action SERVER

insert-http X-Real-IP header-value "%is"

policy-map multi-match PM-443

class NGINX-443-VIP

loadbalance vip inservice

loadbalance policy NGINX-443-POLICY

loadbalance vip icmp-reply

nat dynamic 443 vlan 99

appl-parameter http advanced-options HTTPS

ssl-proxy server SSL-PROXY

policy-map multi-match PM-80

class NGINX-80-VIP

loadbalance vip inservice

loadbalance policy NGINX-80-POLICY

loadbalance vip icmp-reply

nat dynamic 99 vlan 99

appl-parameter http advanced-options HTTP

interface vlan 99

ip address 10.99.1.3 255.255.255.0

peer ip address 10.99.1.5 255.255.255.0

access-group input ALL

access-group output ALL

nat-pool 443 10.99.1.4 10.99.1.4 netmask 255.255.255.255 pat

nat-pool 99 10.99.1.2 10.99.1.2 netmask 255.255.255.255 pat

service-policy input remote_mgmt_allow_policy

service-policy input PM-80

service-policy input PM-443

no shutdown

interface vlan 172

ip address 172.16.1.6 255.255.255.0

peer ip address 172.16.1.12 255.255.255.0

access-group input ALL

service-policy input remote_mgmt_allow_policy

no shutdown

Kanwaljeet Singh · ‎11-11-2013

Hi Vladimir,

Did you try using X-forwarded-for instead of X-real-ip?

insert-http X-Forwarded-For header-value "%is.

The above ensures that you see the client IP in HTTP header and server should have some sort of script enable to catch this for reporting purpose.

I have seen this working flawlessly and would suggest using this and see if that resolves your problem?

Regards,

Kanwal

Kanwaljeet Singh · ‎11-11-2013

HI Vladimir,

I also see in application parameter that you are using header modify per request as well as persistence rebalance so you should see client IP in request forwarded by ACE. I have not particularly used x-real-ip but i have used x-forwarded-for and it works absolutely fine.

Regards,

Kanwal

Vladimir Buyalsky · ‎11-12-2013

Thank you for reply, Fnu!

Unfortunately, I can't to test your advice in near time

Commonly, does Ace support header "x-real-ip"? Could be another reason?

Kanwaljeet Singh · ‎11-12-2013

HI Vladimir,

I checked and ACE should support x-real-ip. The header insertion can be anything but the value should be standard and you are using the right one. So it should work.

So coming to the problem, you say that you see some connections with ACE fowarding it's own IP, rather than client, is that correct?

Are those connections part of the same policy to which you have applied insert-http action?

Regards,

Kanwal

Vladimir Buyalsky · ‎11-13-2013

Hi, Fnu.

Yes, i have only one policy in the config and all connections belong it.

Is there difference between insert-http in the policy and active-list? May be it need to replace active-list?

Kanwaljeet Singh · ‎11-14-2013

Hi Vladimir,

For inserting the client IP you don't need an action list and insert-http X-real-IP header-value "%is" is enough. You can remove the action list that you have configured above and it should still work fine by applying insert -http xxxx ..under policy map.

I am not sure what exactly is the difference but we use insert-http for this scenario.

Regards,

Kanwal

Vladimir Buyalsky · ‎11-14-2013

Hi, Fnu.

I deleted action-list and finally did test of x-forwared-for. It not work, unfortunately. Some packets was arriving with ip of ace in http-header again. (

Kanwaljeet Singh · ‎11-14-2013

Hi Vladimir,

The configuration looks fine. Does this happen to specific connections or is it random? Do you see ACE NAT IP in x-forwarded-for or you don't see anything at all? Can you paste here what you see in HTTP header? Both good and not-good output.

Do you see insert errors increasing under "show stats http"?

Is it happening for HTTP OR HTTPS connections?

Regards,

Kanwal

Vladimir Buyalsky · ‎11-15-2013

Hi, Fnu.

It is happening for HTTP and HTTPS.

Now https is disable and doesn't go through ace.

Header insert errors      : 0                  Max parselen errors      : 29
Static parse errors        : 1216             Resource errors          : 0
Invalid path errors          : 0                  Bad HTTP version errors : 0
Headers rewritten          : 24018818     Header rewrite errors    : 0

I can't find a dependence.

so, there is output of log nginx below, that connect to ace:

good log

example.com 9X.7X.54.200 - [15/Nov/2013:15:07:20 +0400] "GET /path/view/?id=0142200001313011430 HTTP/1.1" 200 43544 "-" Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) 1.000 10.X.X.XX:80 -

bad log

example.com 10.99.1.2 - [15/Nov/2013:14:47:15 +0400] "GET /images/blocks/picture.png HTTP/1.1" 400 0 "-" Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.16 0.100 - -

example.com 10.99.1.2 - [15/Nov/2013:10:21:19 +0400] "GET /pathto/?q=\xD0\xBA\xD1\x81\xD0\xB5\xD1\x84\xD0\xB4.\xEF\xBF\xBD,4/\xEF\xBF\xBD\x18Y\xEF\xBF\xBDOLI\xEF\xBF\xBD\x1A[Z]\x0FML\xEF\xBF\xBD\xDC\x99\x19\x5C\xEF\xBF\xBD\x5C\x1DX\xEF\xBF\xBD\x18]\x19I\xEF\xBF\xBD\x1A\x5C\xEF\xBF\xBDY\x19\x5C\xEF\xBF\xBD\xC9\x8FI\xEF\xBF\xBD\x1E\x5C\x19W\xEF\xBF\xBDO[\xDB\x89\xEF\xBF\xBD\x1E\x5C\x19W\xD8\x8F[\xDB\x89\xEF\xBF\xBD\x1E\x5C\x19W\xEF\xBF\xBD\xEF\xBF\xBD[\xDB\x89\xEF\xBF\xBD\x1E\x5C\x19W\xEF\xBF\xBD HTTP/1.1" 200 18338 "http://X.X.X.X:5080/nonauth/expiration.php?dest=aHR0cDovL2V0cC5yb3NlbHRvcmcucnUvdHJhZGUvcGFzdC8%2FcT3QutGB0LXRhNC%2B0LrQsNC8JnBhZ2U9MSZsaW1pdD01MCZvcmRlcj1wdWJkYXRlJmRpcj1kZXNjJj0mdHlwZV9hPW9uJnR5cGVfYj1vbiZ0eXBlX2M9b24mdHlwZV9wPW9u&expired=1" Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) 3.400 10.X.X.X:80 -

I have changed some IP and domain names.

rajsures · ‎11-15-2013

Hi Vladimir,

Can you please try "persistence-rebalance strict" in the http parameter-map and see if it helps ?

Thanks,

Rajesh.

Vladimir Buyalsky · ‎11-18-2013

Hi, Rajesh.

Thank you for your reply!

Seems, it works, but only partially. So it is very strange. For serverfarm NGINX-FARM-443 all is good, but serverfarm NGINX-FARM-80 keep to receive packet with ip of ace instead of real ip sometimes.

Kanwaljeet Singh · ‎11-19-2013

Hi Vladimir,

Everything looks good but it is weird that it is working for HTTPS and not HTTP. I would suggest opening a TAC case and since this is reproducible it could be new bug or something which we are missing.

As far as configuration goes it is fine.

Regards,

Kanwal