08-03-2012 12:27 PM
With Sivakumar Sukumar
Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about configuration and troubleshooting the Cisco Application Control Engine (ACE) loadbalancer with Sivakumar Sukumar. The Cisco ACE Application Control Engine Module for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series Routers is a next-generation load-balancing and application-delivery solution. A member of the Cisco family of Data Center 3.0 solutions, the module:
Sivakumar Sukumar is an experienced support engineer with the High Touch Technical Support content team, covering all Cisco content delivery network technologies including Cisco Application Control Engine (ACE), Cisco Wide Area Application Services (WAAS), Cisco Content Switching Module, Cisco Content Services Switches, and other content products. He has been with Cisco for more than 2 years, working with major customers to help resolve their issues related to content products. He holds CCNP and DCASI certification.
Remember to use the rating system to let Sivakumar know if you have received an adequate response.
Sivakumar might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum shortly after the event. This event lasts through August 24, 2012. Visit this forum often to view responses to your questions and the questions of other community members.
08-08-2012 07:09 AM
Hi,
what are best steps/commands to look at ACE confguration in regards to troubleshoot issues related to load balancing.
For example - if I know the VIP IP and need to know how many real servers/policies are associated with it just to check things configured for one specific VIP.
Thanks
Ajay
08-08-2012 10:06 AM
Hi Ajay,
There is a handy CLI command that will pull out relevant ACE running-config for a class-map. It looks like it is available on some 3.x version and on 4.x versions and above. In some code versions it is hidden.
show running-config filter [policy-map name] [class-map name]
This will parse the running config for the configuration that is applicable to the policy-map and class-map that is specified.
Regards,
Siva
08-09-2012 07:34 AM
Thanks Siva.. It works for me
08-13-2012 09:51 AM
Hi Siva,
I have a Cisco 6500 and Cisco ACE 4710 with the following configuration/connection.
Cisco 6500's 6/29 (vlan 694 for mgmt) connects to ACE 4710's 1/1
Cisco 6500's 6/30 (vlan 697 for client-side) connects to ACE 4710's 1/2
Cisco 6500's 6/31 (vlan 698 for server-side) connects to ACE 4710's 1/3
***********************Cisco 6500***********************
interface GigabitEthernet6/29
description ACE4710 (Mgmt/Int 1/1)
switchport
switchport access vlan 694
no ip address
no cdp enable
!
interface GigabitEthernet6/30
description ACE4710 (Int 1/2)
switchport
switchport access vlan 697
no ip address
no cdp enable
!
interface GigabitEthernet6/31
description ACE4710 (Int 1/3)
switchport
switchport access vlan 698
no ip address
no cdp enable
!
interface Vlan694
ip address 10.78.2.1 255.255.255.248
interface Vlan697
ip address 10.10.40.1 255.255.255.0
!
interface Vlan698
ip address 10.10.50.1 255.255.255.0
***********************ACE 4710***********************
ACE4710/Admin# show run
Generating configuration....
boot system image:c4710ace-mz.A3_2_1.bin
boot system image:c4710ace-mz.A1_8_0a.bin
hostname ACE4710
interface gigabitEthernet 1/1
switchport access vlan 694
no shutdown
interface gigabitEthernet 1/2
description Client-side
switchport access vlan 697
no shutdown
interface gigabitEthernet 1/3
description Server-side
switchport access vlan 698
no shutdown
interface gigabitEthernet 1/4
shutdown
access-list ALL line 8 extended permit ip any any
class-map type management match-any remote_access
2 match protocol xml-https any
3 match protocol icmp any
4 match protocol telnet any
5 match protocol ssh any
6 match protocol http any
7 match protocol https any
8 match protocol snmp any
policy-map type management first-match remote_mgmt_allow_policy
class remote_access
permit
interface vlan 694
ip address 10.78.2.2 255.255.255.248
access-group input ALL
service-policy input remote_mgmt_allow_policy
no shutdown
interface vlan 697
ip address 10.10.40.2 255.255.255.0
fragment chain 112
access-group input ALL
no shutdown
interface vlan 698
ip address 10.10.50.2 255.255.255.0
fragment chain 112
access-group input ALL
no shutdown
ip route 0.0.0.0 0.0.0.0 10.78.2.1
I will assign other ports with the assicated vlans for client and server on the Cisco 6500. Is it a valid setup/configuration?
If not, what should I change? How to make sure that the client traffic and server traffic can be handled by the ACE 4710? Any suggestion for configuration?
Thanks a lot.
Philip
08-13-2012 11:02 AM
Hi Philip,
Thanks for your question.
The configuration looks good for basic management setup.
Attached the configuration for client to server communication via ACE.
rserver host SERVER_01
ip address 10.10.50.x
inservice
rserver host SERVER_02
ip address 10.10.50.x
inservice
serverfarm host REAL_SERVERS
rserver SERVER_01
inservice
rserver SERVER_02
inservice
class-map match-all VIP
2 match virtual-address 10.10.40.x any
policy-map type loadbalance first-match SLB_LOGIC
class class-default
serverfarm REAL_SERVERS
policy-map multi-match CLIENT_VIPS
class VIP
loadbalance vip inservice
loadbalance policy SLB_LOGIC
loadbalance vip icmp-reply active
interface vlan 697
service-policy input CLIENT_VIPS
no shutdown
Also here is a guide to setup basic server loadbalancing with step by step configuration.
Let me know if you have any questions.
Regards,
Siva
08-13-2012 04:07 PM
First, thank you for the quick note, Siva.
However, I am not able to browse/ping the VIP web server from client now.
VIP: 10.10.40.2
Real Web Server IP: 10.10.50.3
Client IP on vlan 697: 10.10.40.3
Here is what I have now.
***********************ACE 4710***********************
hostname ACE4710
interface gigabitEthernet 1/1
switchport access vlan 694
no shutdown
interface gigabitEthernet 1/2
description Client-side
switchport access vlan 697
no shutdown
interface gigabitEthernet 1/3
description Server-side
switchport access vlan 698
no shutdown
interface gigabitEthernet 1/4
shutdown
access-list ALL line 8 extended permit ip any any
probe http 1
interval 15
passdetect interval 60
request method get url http://10.10.50.3
open 10
rserver host SERVER_01
ip address 10.10.50.3
conn-limit max 4000000 min 4000000
inservice
rserver host SERVER_02
ip address 10.10.50.4
conn-limit max 4000000 min 4000000
inservice
serverfarm host REAL_SERVERS
probe 1
rserver SERVER_01 80
conn-limit max 4000000 min 4000000
inservice
rserver SERVER_02 80
conn-limit max 4000000 min 4000000
inservice
class-map match-all VIP
2 match virtual-address 10.10.40.20 any
class-map match-all VIP2
2 match virtual-address 10.10.40.20 tcp eq www
class-map type management match-any remote_access
2 match protocol xml-https any
3 match protocol icmp any
4 match protocol telnet any
5 match protocol ssh any
6 match protocol http any
7 match protocol https any
8 match protocol snmp any
policy-map type management first-match remote_mgmt_allow_policy
class remote_access
permit
policy-map type loadbalance first-match SLB_LOGIC
class class-default
serverfarm REAL_SERVERS
policy-map type loadbalance first-match VIP2-l7slb
class class-default
serverfarm REAL_SERVERS
policy-map multi-match CLIENT_VIPS
class VIP
loadbalance vip inservice
loadbalance policy SLB_LOGIC
loadbalance vip icmp-reply active
class VIP2
loadbalance vip inservice
loadbalance policy VIP2-l7slb
interface vlan 694
ip address 10.78.2.2 255.255.255.248
access-group input ALL
service-policy input remote_mgmt_allow_policy
no shutdown
interface vlan 697
ip address 10.10.40.2 255.255.255.0
fragment chain 112
access-group input ALL
service-policy input CLIENT_VIPS
no shutdown
interface vlan 698
ip address 10.10.50.2 255.255.255.0
fragment chain 112
access-group input ALL
no shutdown
ip route 0.0.0.0 0.0.0.0 10.78.2.1
Thanks.
Philip
08-13-2012 10:32 PM
Hi Philip,
Are you able to ping the gateway, servers and 6500 SVI's from the ACE? Can you send me the output of "show service-policy detail" & "show arp"?
Regards,
Siva
08-14-2012 09:56 AM
Hi Siva,
With the above configuration, I can ping server-side's gateway (IP: 10.10.50.1) and client-side's gateway (IP: 10.10.40.1) from ACE. However, I can't ping (IP: 10.10.50.2) and (IP: 10.10.40.2) from 6500. I thought the icmp is allowed from the above configuration.
ACE4710/Admin# show service-policy detail
Policy-map : CLIENT_VIPS
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 1 697
service-policy: CLIENT_VIPS
class: VIP
VIP Address: Protocol: Port:
10.10.40.20 any
loadbalance:
L7 loadbalance policy: SLB_LOGIC
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP state: OUTOFSERVICE
Persistence Rebalance: DISABLED
curr conns : 0 , hit count : 24
dropped conns : 24
client pkt count : 36 , client byte count: 1728
server pkt count : 0 , server byte count: 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : SLB_LOGIC
class/match : class-default
LB action :
primary serverfarm: REAL_SERVERS
state: DOWN
backup serverfarm : -
hit count : 24
dropped conns : 0
compression : off
compression:
bytes_in : 0
bytes_out : 0
Compression ratio : 0.00%
class: VIP2
VIP Address: Protocol: Port:
10.10.40.20 tcp eq 80
loadbalance:
L7 loadbalance policy: VIP2-l7slb
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP state: OUTOFSERVICE
Persistence Rebalance: DISABLED
curr conns : 0 , hit count : 0
dropped conns : 0
client pkt count : 0 , client byte count: 0
server pkt count : 0 , server byte count: 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : VIP2-l7slb
class/match : class-default
LB action :
primary serverfarm: REAL_SERVERS
state: DOWN
backup serverfarm : -
hit count : 0
dropped conns : 0
compression : off
compression:
bytes_in : 0
bytes_out : 0
Compression ratio : 0.00%
ACE4710/Admin# show arp
Context Admin
================================================================================
IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status
================================================================================
10.78.2.1 00.12.da.10.3c.0a vlan694 GATEWAY 5 38 sec up
10.78.2.2 00.1b.24.3d.bc.8c vlan694 INTERFACE LOCAL _ up
10.10.40.1 00.12.da.10.3c.0a vlan697 LEARNED 8 10629 sec up
10.10.40.2 00.1b.24.3d.bc.8c vlan697 INTERFACE LOCAL _ up
10.10.40.3 00.50.56.94.2f.ba vlan697 LEARNED 7 10589 sec up
10.10.40.20 00.1b.24.3d.bc.8c vlan697 VSERVER LOCAL _ up
10.10.50.1 00.12.da.10.3c.0a vlan698 LEARNED 9 4601 sec up
10.10.50.2 00.1b.24.3d.bc.8c vlan698 INTERFACE LOCAL _ up
10.10.50.3 00.50.56.93.00.db vlan698 RSERVER 10 235 sec up
10.10.50.4 00.00.00.00.00.00 vlan698 RSERVER - * 1 req dn
================================================================================
Total arp entries 10
Thanks.
Philip
08-14-2012 10:10 AM
Hi Philip,
The VIP state is OUTOFSERVICE.
Can you remove the probe from serverfarm and check if the VIP changes to INSERVICE and serverfarm comes UP?
serverfarm host REAL_SERVERS
probe 1 <<<<<<<<<<<<<< REMOVE>>>>>>>>>>>>>>
VIP state: OUTOFSERVICE <<<<<<<<<<<<<<<
Persistence Rebalance: DISABLED
curr conns : 0 , hit count : 24
dropped conns : 24
client pkt count : 36 , client byte count: 1728
server pkt count : 0 , server byte count: 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : SLB_LOGIC
class/match : class-default
LB action :
primary serverfarm: REAL_SERVERS
state: DOWN <<<<<<<<<<<<<<<<<<<<
Once VIP is INSERVICE check if you can ping the VIP - 10.10.40.20 from 6500.
If it still shows OUTOFSERVICE after removing the probe send me the output of "show serverfarm detail" & "show rserver detail"
Also can you check if you are able to ping 10.78.2.2 from 6500?
Regards,
Siva
08-14-2012 10:46 AM
Hi Siva,
I can ping VIP (IP: 10.10.40.20) from 6500, also from the client-side (IP: 10.10.40.3), but I still can't browse from the VIP for both webservers when I tried http://10.10.40.20 from the client's web browser. I even see reset from the pcap.
I can browse both webservers direct without an issue though.
ACE4710/Admin# show service-policy detail
Policy-map : CLIENT_VIPS
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 1 697
service-policy: CLIENT_VIPS
class: VIP
VIP Address: Protocol: Port:
10.10.40.20 any
loadbalance:
L7 loadbalance policy: SLB_LOGIC
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP State: INSERVICE
Persistence Rebalance: DISABLED
curr conns : 0 , hit count : 28
dropped conns : 28
client pkt count : 42 , client byte count: 2016
server pkt count : 0 , server byte count: 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : SLB_LOGIC
class/match : class-default
LB action :
primary serverfarm: REAL_SERVERS
state: UP
backup serverfarm : -
hit count : 28
dropped conns : 0
compression : off
compression:
bytes_in : 0
bytes_out : 0
Compression ratio : 0.00%
class: VIP2
VIP Address: Protocol: Port:
10.10.40.20 tcp eq 80
loadbalance:
L7 loadbalance policy: VIP2-l7slb
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP State: INSERVICE
Persistence Rebalance: DISABLED
curr conns : 0 , hit count : 0
dropped conns : 0
client pkt count : 0 , client byte count: 0
server pkt count : 0 , server byte count: 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : VIP2-l7slb
class/match : class-default
LB action :
primary serverfarm: REAL_SERVERS
state: UP
backup serverfarm : -
hit count : 0
dropped conns : 0
compression : off
compression:
bytes_in : 0
bytes_out : 0
Compression ratio : 0.00%
ACE4710/Admin# show serverfarm detail
serverfarm : REAL_SERVERS, type: HOST
total rservers : 2
active rservers: 2
description : -
state : ACTIVE
predictor : ROUNDROBIN
failaction : -
back-inservice : 0
partial-threshold : 0
num times failover : 5
num times back inservice : 7
total conn-dropcount : 0
---------------------------------
----------connections-----------
real weight state current total failures
---+---------------------+------+------------+----------+----------+---------
rserver: SERVER_01
10.10.50.3:80 8 OPERATIONAL 0 0 18
max-conns : 4000000 , out-of-rotation count : 0
min-conns : 4000000
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
retcode out-of-rotation count : -
rserver: SERVER_02
10.10.50.4:80 8 OPERATIONAL 0 0 0
max-conns : 4000000 , out-of-rotation count : 0
min-conns : 4000000
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
retcode out-of-rotation count : -
ACE4710/Admin# show rserver detail
rserver : SERVER_01, type: HOST
state : OPERATIONAL (verified by arp response)
description : -
max-conns : 4000000 , out-of-rotation count : 0
min-conns : 4000000
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
weight : 8
---------------------------------
----------connections-----------
real weight state current total
---+---------------------+------+------------+----------+--------------------
serverfarm: REAL_SERVERS
10.10.50.3:80 8 OPERATIONAL 0 0
max-conns : 4000000 , out-of-rotation count : 0
min-conns : 4000000
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
total conn-failures : 18
rserver : SERVER_02, type: HOST
state : OPERATIONAL (verified by arp response)
description : -
max-conns : 4000000 , out-of-rotation count : 0
min-conns : 4000000
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
weight : 8
---------------------------------
----------connections-----------
real weight state current total
---+---------------------+------+------------+----------+--------------------
serverfarm: REAL_SERVERS
10.10.50.4:80 8 OPERATIONAL 0 0
max-conns : 4000000 , out-of-rotation count : 0
min-conns : 4000000
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
total conn-failures : 0
Thanks.
Philip
08-14-2012 11:11 AM
Hi Philip,
Was this capture taken from client? If so can you apply NAT on ACE and see if it works.
policy-map multi-match CLIENT_VIPS
class VIP
loadbalance vip inservice
loadbalance policy SLB_LOGIC
loadbalance vip icmp-reply active
nat dynamic 1 vlan 698 <<<<<<<<<<< ADD >>>>>>>>>>>.
interface vlan 698
nat-pool 1 10.10.50.10 10.10.50.10 netmask 255.255.255.255 pat <<<<<<<<<<< ADD >>>>>>>>>>>.
Regards,
Siva
08-14-2012 11:39 AM
Hi Siva,
Yes the pcap is from the client. It works now. Could you explain a little bit of the reason why we need the nat here?
We have the client(IP: 10.10.40.3) sending GET a resquest to ACE's VIP (IP: 10.10.40.20). ACE's VIP goes to real servers (IP: 10.10.50.3, 10.10.50.4). Real servers reply to the nat (IP: 10.10.50.10) and maps it back to the VIP -> client?
Where is the NAT (IP: 10.10.50.10) playing? What's the logic?
Thanks.
Philip
08-14-2012 11:54 AM
Hi Philip,
That's correct.
The problem was due to asymmetric routing and the server replies directly back to the client bypassing ACE.
The trick here is getting return traffic from the real server to go back through the ACE; this is achieved with source NAT. We create a NAT pool on the ACE and when the user hits the ACE, his address is translated to one in the pool. The real server sees the source address as one in the pool and knows that that subnet resides on the ACE, so server replies to the ACE. The ACE then NATs the address to the user’s real address and forwards the response.
Another option is to change the routing on server so it always responds backs to ACE instead of replying directly back to the client.
Regards,
Siva
08-14-2012 03:04 PM
Thank you for the information, Siva.
So does it mean putting ip route 10.10.50.0 255.255.255.0 10.10.50.2 on the 6500 will take care of it then?
Thanks.
Philip
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide