ACE 4710 no real ip in header with referer

[Vladimir Buyalsky]

Hello everybody. I need help with ACE 4710. We use one-arm PAT config to round robin load balance scheme with ssl terminate and trying to insert x-real-ip in all http-headers. But some packets with referer inside arrive with ACE ip in header, no real ip of clients. Any ideas, please?

Thanks!

Config ACE:

interface gigabitEthernet 1/1

  channel-group 1

  no shutdown

interface gigabitEthernet 1/2

  channel-group 1

  no shutdown

interface gigabitEthernet 1/3

  channel-group 1

  no shutdown

interface gigabitEthernet 1/4

  channel-group 1

  no shutdown

interface port-channel 1

  switchport trunk native vlan 1

  switchport trunk allowed vlan 2-200

  no shutdown

crypto chaingroup intercert

access-list ALL line 8 extended permit ip any any

access-list ALL line 9 extended permit icmp any any

probe icmp ICMP_PROBE

  interval 10

  passdetect count 4

  receive 1

probe http http_probe

  interval 5

  passdetect interval 10

  passdetect count 2

  expect status 200 210

  header User-Agent header-value "LoadBalance"

probe http http_probe_443

  port 443

  interval 5

  passdetect interval 10

  passdetect count 2

  expect status 200 210

  header User-Agent header-value "LoadBalance"

rserver host nginx-1

  ip address 10.99.1.11

  inservice

rserver host nginx-2

  ip address 10.99.1.12

  inservice

rserver host nginx-3

  ip address 10.99.1.13

  inservice

rserver host nginx-4

  ip address 10.99.1.14

  inservice

rserver host nginx-5
  ip address 10.99.1.15
  inservice

rserver host nginx-6

  ip address 10.99.1.16

  inservice

rserver host nginx-7

  ip address 10.99.1.17

  inservice

rserver host nginx-8

  ip address 10.99.1.18

  inservice

serverfarm host NGINX-FARM-443

  probe ICMP_PROBE

  probe http_probe_443

  rserver nginx-5

    inservice

serverfarm host NGINX-FARM-80

  probe ICMP_PROBE

  probe http_probe

  rserver nginx-1

    inservice

  rserver nginx-2

    inservice

parameter-map type http HTTP

  case-insensitive

  persistence-rebalance

  header modify per-request

  set header-maxparse-length 16384

  length-exceed continue

parameter-map type http HTTPS

  persistence-rebalance

  header modify per-request

  set header-maxparse-length 16384

  length-exceed continue

sticky ip-netmask 255.255.255.255 address source STUCK

  serverfarm NGINX-FARM-80

  timeout 60

  replicate sticky

sticky ip-netmask 255.255.255.255 address source STUCK443

  serverfarm NGINX-FARM-443

  timeout 60

  replicate sticky

action-list type modify http SERVER

  header rewrite response server header-value "nginx" replace "hamster"

action-list type modify http REAL-SSL

  header insert request x-real-ip header-value "%is"

ssl-proxy service SSL-PROXY

  key xxx.em

  cert xxx.pem

class-map match-all NGINX-443-VIP

  2 match virtual-address 10.99.1.4 tcp eq https

class-map match-all NGINX-80-VIP

  2 match virtual-address 10.99.1.2 tcp eq www

class-map type management match-any remote_access

  2 match protocol xml-https any

  3 match protocol icmp any

  4 match protocol telnet any

  5 match protocol ssh any

  6 match protocol http any

  7 match protocol https any

  8 match protocol snmp any

policy-map type management first-match remote_mgmt_allow_policy

  class remote_access

    permit

policy-map type loadbalance http first-match NGINX-443-POLICY

  class class-default

    sticky-serverfarm STUCK443

    insert-http X-Real-IP header-value "%is"

policy-map type loadbalance http first-match NGINX-80-POLICY

  class class-default

    sticky-serverfarm STUCK

    action SERVER

    insert-http X-Real-IP header-value "%is"

policy-map multi-match PM-443

  class NGINX-443-VIP

    loadbalance vip inservice

    loadbalance policy NGINX-443-POLICY

    loadbalance vip icmp-reply

    nat dynamic 443 vlan 99

    appl-parameter http advanced-options HTTPS

    ssl-proxy server SSL-PROXY

policy-map multi-match PM-80

  class NGINX-80-VIP

    loadbalance vip inservice

    loadbalance policy NGINX-80-POLICY

    loadbalance vip icmp-reply

    nat dynamic 99 vlan 99

    appl-parameter http advanced-options HTTP

interface vlan 99

  ip address 10.99.1.3 255.255.255.0

  peer ip address 10.99.1.5 255.255.255.0

  access-group input ALL

  access-group output ALL

  nat-pool 443 10.99.1.4 10.99.1.4 netmask 255.255.255.255 pat

  nat-pool 99 10.99.1.2 10.99.1.2 netmask 255.255.255.255 pat

  service-policy input remote_mgmt_allow_policy

  service-policy input PM-80

  service-policy input PM-443

  no shutdown

interface vlan 172

  ip address 172.16.1.6 255.255.255.0

  peer ip address 172.16.1.12 255.255.255.0

  access-group input ALL

  service-policy input remote_mgmt_allow_policy

  no shutdown

[Kanwaljeet Singh]

Hi Vladimir,

Did you try using X-forwarded-for instead of X-real-ip?

insert-http X-Forwarded-For header-value "%is.

The above ensures that you see the client IP  in HTTP header and server should have some sort of script enable to catch this for reporting purpose.

I have seen this working flawlessly and would suggest using this and see if  that resolves your problem?

Regards,

Kanwal

[Kanwaljeet Singh]

HI Vladimir,

I also see in application parameter that you are using header modify per request as well as persistence rebalance so you should see client IP in request forwarded by ACE. I have not particularly used x-real-ip but i have used x-forwarded-for and it works absolutely fine.

Regards,

Kanwal

[Vladimir Buyalsky]

Thank you for reply, Fnu!

Unfortunately, I can't to test your advice in near time

Commonly, does Ace support header "x-real-ip"? Could be another reason?

[Kanwaljeet Singh]

HI Vladimir,

I checked and ACE should support x-real-ip. The header insertion can be anything but the value should be standard and you are using the right one. So it should work.

So coming to the problem, you say that you see some connections with ACE fowarding it's own IP, rather than client, is that correct?

Are those connections part of the same policy to which you have applied insert-http action?

Regards,

Kanwal

[Vladimir Buyalsky]

Hi, Fnu.

Yes, i have only one policy in the config and all connections belong it.

Is there difference between insert-http in the policy and active-list? May be it need to replace active-list?

[Kanwaljeet Singh]

Hi Vladimir,

For inserting the client IP you don't need an action list and insert-http X-real-IP header-value "%is" is enough.  You can remove the action list that you have configured above and it should still work fine by applying  insert -http xxxx ..under policy map.

I am not sure what exactly is the difference but we use insert-http for this scenario.

Regards,

Kanwal

[Vladimir Buyalsky]

Hi, Fnu.

I deleted action-list and finally did test of x-forwared-for. It not work, unfortunately. Some packets was arriving with ip of ace in http-header again. (

[Kanwaljeet Singh]

Hi Vladimir,

The configuration looks fine.  Does this happen to specific connections or is it random? Do you see ACE NAT IP in x-forwarded-for or you don't see anything at all? Can you paste here what you see in HTTP header? Both good and not-good output.

Do you see insert errors increasing under "show stats http"?

Is it happening for HTTP OR HTTPS connections?

Regards,

Kanwal

[Vladimir Buyalsky]

Hi, Fnu.

It is happening for HTTP and HTTPS.

Now https is disable and doesn't go through ace.

Header insert errors      : 0                  Max parselen errors      : 29       
Static parse errors        : 1216             Resource errors          : 0        
Invalid path errors          : 0                  Bad HTTP version errors  : 0        
Headers rewritten          : 24018818     Header rewrite errors    : 0        

I can't find a dependence.

so, there is output of log nginx below, that connect to ace:

good log

example.com 9X.7X.54.200 - [15/Nov/2013:15:07:20 +0400] "GET /path/view/?id=0142200001313011430 HTTP/1.1" 200 43544 "-" Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) 1.000 10.X.X.XX:80 -

bad log

example.com 10.99.1.2 - [15/Nov/2013:14:47:15 +0400] "GET /images/blocks/picture.png HTTP/1.1" 400 0 "-" Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.16 0.100 - -

example.com 10.99.1.2 - [15/Nov/2013:10:21:19 +0400] "GET /pathto/?q=\xD0\xBA\xD1\x81\xD0\xB5\xD1\x84\xD0\xB4.\xEF\xBF\xBD,4/\xEF\xBF\xBD\x18Y\xEF\xBF\xBDOLI\xEF\xBF\xBD\x1A[Z]\x0FML\xEF\xBF\xBD\xDC\x99\x19\x5C\xEF\xBF\xBD\x5C\x1DX\xEF\xBF\xBD\x18]\x19I\xEF\xBF\xBD\x1A\x5C\xEF\xBF\xBDY\x19\x5C\xEF\xBF\xBD\xC9\x8FI\xEF\xBF\xBD\x1E\x5C\x19W\xEF\xBF\xBDO[\xDB\x89\xEF\xBF\xBD\x1E\x5C\x19W\xD8\x8F[\xDB\x89\xEF\xBF\xBD\x1E\x5C\x19W\xEF\xBF\xBD\xEF\xBF\xBD[\xDB\x89\xEF\xBF\xBD\x1E\x5C\x19W\xEF\xBF\xBD HTTP/1.1" 200 18338 "http://X.X.X.X:5080/nonauth/expiration.php?dest=aHR0cDovL2V0cC5yb3NlbHRvcmcucnUvdHJhZGUvcGFzdC8%2FcT3QutGB0LXRhNC%2B0LrQsNC8JnBhZ2U9MSZsaW1pdD01MCZvcmRlcj1wdWJkYXRlJmRpcj1kZXNjJj0mdHlwZV9hPW9uJnR5cGVfYj1vbiZ0eXBlX2M9b24mdHlwZV9wPW9u&expired=1" Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) 3.400 10.X.X.X:80 -

I have changed some IP and domain names.

[rajsures]

Hi Vladimir,

Can you please try "persistence-rebalance strict" in the http parameter-map and see if it helps ?

Thanks,

Rajesh.

[Vladimir Buyalsky]

Hi, Rajesh.

Thank you for your reply!

Seems, it works, but only partially. So it is very strange. For serverfarm NGINX-FARM-443 all is good, but serverfarm NGINX-FARM-80 keep to receive packet with ip of ace instead of real ip sometimes.

[Kanwaljeet Singh]

Hi Vladimir,

Everything looks good but it is weird that it is working for HTTPS and not HTTP. I would suggest opening a TAC case and since this is reproducible it could be new bug or something which we are missing.

As far as configuration goes it is fine.

Regards,

Kanwal