Why ACE in Bridge Mode needs L3 routes ?

[luismatos99]

Hi,

I was reading an old discussion (https://supportforums.cisco.com/message/461263#461263) and the person said that his ACE only starts work after he included a default route in the context even if it was in bridge mode configuration. I am using an equal scenario that have multiple BVI interfaces.

I am facing an equal situation, I have configured an ACE module using bridge mode and some client sources does not work using the VIP address to access port 8080. This problem occurs only with some client source networks the rest of them work fine. Since we are using bridge mode there was only a default route (in each context) pointed to the management vlan (vlan 40) to use TACACS, SYSLOG, ETC ...

The source networks that can not access the VIP using port 8080 can do it:

1) ping the VIP without any problem

2) access the real server ip address using port 8080 without any problem

The problem occur only when these specific sources try to access the VIP address using port 8080.

Yesterday a very strange behavior ocurred: The client source networks that were not able to access the VIP using port 8080 start to work after I created some specific routes destinated to them using another different NEXT HOP ip address address. After this routes were included the problem was solved.

My doubts :

1) Why a L3 information is changing the ACE behavior in a bridge mode?

2) Can I solve this problem using "mac-sticky enable" in the client vlan interfaces instead of the specific routes ? In case positive, why ?

3) Does ACE use L3 information to create the flow table even if configured in bridge mode?

Clients are using this VIP : "match virtual-address 10.128.1.70 tcp eq 8080"

VLAN 401  = CLIENT SIDE

VLAN1401 = SERVER SIDE

This is the configuration that we are talking about:

======================================================================================

access-list acl_permit line 8 extended permit ip any any

probe http http_probe
  interval 5
  passdetect interval 10
  expect status 200 200
probe https https_probe
  interval 5
  passdetect interval 10
  expect status 200 200
probe tcp porta-1080_probe
  port 1080
  interval 5
  passdetect interval 10
probe tcp porta-8080_probe
  port 8080
  interval 5
  passdetect interval 10
probe http porta-proxy_probe
  port 8080
  interval 5
  passdetect interval 10
  expect status 200 200

rserver host server79
  description BARRAS
  ip address 10.128.184.56
  inservice
rserver host server80
  description ITATINGA
  ip address 10.128.184.52
  inservice
rserver host server87
  description DANCADOR
  ip address 10.128.1.88
  inservice
rserver host server88
  description DANCARINO
  ip address 10.128.1.89
  inservice
rserver host server89
  description DIABLOTIM
  ip address 10.128.1.90
  inservice
rserver host server96
  description AREAL
  ip address 10.128.5.68
  inservice
rserver host server97
  description AREIAS
  ip address 10.128.5.69
  inservice

serverfarm host grupo-64
  failaction purge
  predictor leastconns
  probe http_probe
  rserver server79
    inservice
  rserver server80
    inservice
serverfarm host grupo-68
  failaction purge
  predictor hash address
  probe porta-8080_probe
  rserver server87
    inservice
  rserver server88
    inservice
  rserver server89
    inservice
serverfarm host grupo-69
  failaction purge
  predictor hash address
  probe porta-1080_probe
  rserver server87
    inservice
  rserver server88
    inservice
  rserver server89
    inservice
serverfarm host grupo-74
  failaction purge
  probe http_probe
  rserver server96
    inservice
  rserver server97
    inservice

class-map type management match-any remote-access
  2 match protocol https any
  3 match protocol icmp any
  4 match protocol telnet any
  5 match protocol ssh any
  6 match protocol http any
  7 match protocol snmp any
class-map match-any vlan209-vip-class
  5 match virtual-address 10.128.184.57 tcp eq www
class-map match-any vlan401-1080-vip-class
  5 match virtual-address 10.128.1.70 tcp eq 1080
class-map match-any vlan401-8080-vip-class
  5 match virtual-address 10.128.1.70 tcp eq 8080
class-map match-any vlan405-vip-class
  5 match virtual-address 10.128.5.83 tcp eq www

policy-map type management first-match remote-mgmt
  class remote-access
    permit

policy-map type loadbalance http first-match vlan209-slb-policy
  class class-default
    serverfarm grupo-64
policy-map type loadbalance http first-match vlan401-1080-slb-policy
  class class-default
    serverfarm grupo-69
policy-map type loadbalance http first-match vlan401-8080-slb-policy
  class class-default
    serverfarm grupo-68
policy-map type loadbalance http first-match vlan405-slb-policy
  class class-default
    serverfarm grupo-74

policy-map multi-match vlan209-vip-policy
  class vlan209-vip-class
    loadbalance vip inservice
    loadbalance policy vlan209-slb-policy
    loadbalance vip icmp-reply
policy-map multi-match vlan401-vip-policy
  class vlan401-8080-vip-class
    loadbalance vip inservice
    loadbalance policy vlan401-8080-slb-policy
    loadbalance vip icmp-reply
  class vlan401-1080-vip-class
    loadbalance vip inservice
    loadbalance policy vlan401-1080-slb-policy
    loadbalance vip icmp-reply
policy-map multi-match vlan405-vip-policy
  class vlan405-vip-class
    loadbalance vip inservice
    loadbalance policy vlan405-slb-policy
    loadbalance vip icmp-reply

interface vlan 40
  description Gerencia
  ip address 10.128.140.182 255.255.255.224
  peer ip address 10.128.140.168 255.255.255.224
  service-policy input remote-mgmt
  no shutdown
interface vlan 401
  description "Client Side - VLAN 401"
  bridge-group 401
  access-group input acl_permit
  service-policy input remote-mgmt
  service-policy input vlan401-vip-policy
  no shutdown
interface vlan 405
  description "Client Side - VLAN 400"
  bridge-group 405
  access-group input BPDU
  access-group input acl_permit
  service-policy input remote-mgmt
  service-policy input vlan405-vip-policy
  no shutdown
interface vlan 905
  description "Server Side - VLAN405"
  bridge-group 405
  access-group input BPDU
  access-group input acl_permit
  service-policy input remote-mgmt
  no shutdown
interface vlan 1401
  description "Server Side - VLAN401"
  bridge-group 401
  access-group input acl_permit
  service-policy input remote-mgmt
  no shutdown

interface bvi 401
  ip address 10.128.1.210 255.255.255.0
  peer ip address 10.128.1.211 255.255.255.0
  description "VLAN401<->VLAN1401 Bridge Group"
  no shutdown
interface bvi 405
  ip address 10.128.5.210 255.255.255.0
  peer ip address 10.128.5.211 255.255.255.0
  description "VLAN405<->VLAN1405 Bridge Group"
  no shutdown

===== This is the default gateway used for management =====

ip route 0.0.0.0 0.0.0.0 10.128.140.190

===== This is the specific routes that I have included to correct the problem =====

ip route 10.128.168.194 255.255.255.255 10.128.1.254
ip route 10.129.138.0 255.255.255.0 10.128.1.254
ip route 10.128.9.101 255.255.255.255 10.128.1.254
ip route 10.128.9.100 255.255.255.255 10.128.1.254
ip route 10.128.9.102 255.255.255.255 10.128.1.254
ip route 10.128.14.245 255.255.255.255 10.128.1.254
ip route 10.128.201.66 255.255.255.255 10.128.1.254
ip route 10.128.9.62 255.255.255.255 10.128.1.254

*** Note that this route ponted to a different gateway, this different gateway belongs to the VLAN401 (Client side) ****

Thanks for you help !!!

LMatos

[Gilles Dufour]

ACE uses encapid to setup connections.

The encapid represents a known arp entry.

The known arp entries are the one for which we sent directly an arp request - like rserver or gateways.

If we do not have an encapid, we can't establish the connection and drop the traffic.

So, by configuring a gateway, you force ACE to learn the mac-address of the gateway and associate with it a new encapid.

Suddenly, all connections from that source mac-address can access vip.

The ping to the vip is different because we do not setup a flow, but instead just respond back to the client.

I don't see why only vip 8080 was not working.  It's either no vip works or all of them.

Also the ACE can work like a proxy and terminate connections.

In this case, it also needs a routing table in order to decide how to respond to the client.

Gilles.

[luismatos99]

Tks for your attention !! Let me clarify your doubt: We have not tested the another VIPs because they are not available, I agree you that this problem will occur with the others services too.

My doubt is: How can we address this behavior in Bridge mode with multiples BVIs ? I have face that if we configure the default gateway to a next hop that can not reach the source address the session can not be  established even if we are using bridge mode. My old understanding was that the default route (in bridge mode) was used only for management and not to reach the sources located in the client side vlans.

For instance:

I have made this test using only one default route ponited to the management vlan (vlan40) and tried to established a connection with the source 10.128.168.94:

sh conn | inc 10.128.168.94

1522832    1  in  TCP   401  10.128.168.94:3135    10.128.1.70:8080      SYNSEEN
1986333    1  out TCP   1401 10.128.1.89:8080      10.128.168.94:3135    SYNACK
1574802    1  in  TCP   401  10.128.168.94:3131    10.128.1.70:8080      SYNSEEN
2690729    1  out TCP   1401 10.128.1.89:8080      10.128.168.94:3131    SYNACK

As you can see I faced the problem.

But, wen I included a specific route to the source (10.128.168.94) pointed to the default-gateway located in the client vlan (same network range that the interface BVI belongs to) the problem was solved:

ip route 10.128.168.94 255.255.255.255 10.128.1.254

sh conn | inc 10.128.168.94
2564811    2  in  TCP   401  10.128.168.94:3159    10.128.1.70:8080      ESTAB
1571087    2  out TCP   1401 10.128.1.89:8080      10.128.168.94:3159    ESTAB
2092354    2  in  TCP   401  10.128.168.94:3157    10.128.1.70:8080      ESTAB
1641491    2  out TCP   1401 10.128.1.89:8080      10.128.168.94:3157    ESTAB

The main doubt is that this problem not occurs with the entire network, I think that it occurs only when the NEXT HOP of the default route has not a L3 route to the client source address. I tried to implement "mac-sticky" yesterday and I faced that the problem was solved for many client destination but not for some clients IP address located in the same subnet.

For instance:

******** 1st Test: ********

ACE Context with mac-sticky enable on the client vlan (specific route not configured, only default route to the management vlan [10.128.140.190]):

Int vlan401

     mac-sticky enable

     exit

Results:

IP 10.128.9.66 = Works fine

IP 10.128.9.62 = Not Work

With mac-sticky enable I tried to creat a specific route to de source 10.128.9.62:

ip route 10.128.9.62 255.255.255.255 10.128.1.254

But I did not established the session until I removed the mac-stiky enable in the client side interface.

Before I remove the mac-sticky and created a specific route to the source 10.128.9.62:  only source 10.128.9.62 started work, the source 10.128.9.66  did not work anymore.

******** 2nd Test: ********

ACE Context **without** mac-sticky enable on the client vlan and default route changed to the Client vlan default gateway [10.128.1.254]:

no ip route 0.0.0.0 0.0.0.0 10.128.140.190

ip route 0.0.0.0 0.0.0.0 10.128.1.254

IP 10.128.9.66 = Works fine

IP 10.128.9.62 = Works fine

Thank you very much !

LMatos

[Gilles Dufour]

You should also configure " mac-sticky enable" under each interface.

In case there are multiple routes to the same destination, it guarantees that the route we select is the same one traffic came in.

Gilles.