Showing results for 
Search instead for 
Did you mean: 

Datacenter troubleshooting guide - day 2

Gilles Dufour
Cisco Employee

“Datacenter troubleshooting guide” – a blog by Gilles Dufour.

Day 2 – Basic Loadbalancing

Now, let’s focus on basic loadbalancing.  Since you have read the previous post (“Day 1 – Start with the basic) you already did preliminary work by checking connectivity between client – ACE and ACE – server.  Everything is good and you can confirm traffic from client does it the ACE device.

Still, your loadbalancing solution does not work.

The next step is to check if ACE does count the client request against the appropriate VIP.  For that you will use the command show service-policy <policy_name>.

switch/Admin# show service-policy SLB

Status     : ACTIVE


Interface: vlan 10 20

  service-policy: SLB

    class: VIP-122-80
        L7 loadbalance policy: l7-vmware-8080
        VIP Route Metric     : 62
        VIP Route Advertise  : ENABLED-WHEN-ACTIVE
        VIP ICMP Reply       : DISABLED
        VIP State: INSERVICE
        curr conns       : 0         , hit count        : 17
        dropped conns    : 3
        client pkt count : 24        , client byte count: 960
        server pkt count : 24        , server byte count: 1040
        conn-rate-limit      : 0         , drop-count : 0
        bandwidth-rate-limit : 0         , drop-count : 0

Check if the   client pkt countcounter is increasing.  Also check the “server pkt count”. 
If you do not see the server packet counter incrementing, it is often indication of asymmetric routing.  You can quickly confirm this by configuring client nat.  If that solves the problem, it does confirm that you have asymmetric routing.  You then need to either fix the server routing table or keep client nat.

If your client packet counter does not increment and the “hit count” counter also stays unchanged, the traffic does not match this rule or it is not getting to the ACE device.

A quick way to check the traffic hits the correct rule is to call the following command :

show np 1 access-list trace vlan <inbound vlan> in pro <protocol> source <src ip> <src port> destination <dst ip> <dst port>

You can usually ignore the first part of the output from this command.  It begins to be interesting from the “version+aceid”.

In our case the client ip address is  The source port is unknown so I use the value "0".  The virtual ip address is on port 80. Vlan 20 is the interface on which client traffic is coming into my ACE.

switch/Admin# show np 1 access-list trace vlan 20 in pro 6 source 0 des 80


Context ID: 0

<… ignore first part …>

version+aceid 0x16c (version 0 ace_id 364 dirty no)

action_flag 0x2 (permit yes log no punt_to_cp no capture no bridge no)

path ID 0x0

src nat 0x0 dst nat 0x0 vserver 0x51 fixup 0x0

TCP conn 0x0 AAA 0x0 Websense 0x0 QOS Policer 0x0

Syslog Info 0

Hitcount 0

Syslog info:

  idx:[364:0] name_idx:[0:0] hash1:0x0 hash2:0x0 name_len:0 invalid

Number of DRAM access: 16 (5 mtrie 11 non-mtrie)

What you want to look at first is the vserver id.  In our example, the id is 0x51 or 81 in decimal. 

You can then check if it corresponds to your policy-map and class-map by checking the config manager tables.

switch/Admin# show cfgmgr internal table l3-rule | i 81

81              135             104             0       0      DATA_VALID,


This command gives us a line matching our rule 81 and this rule is from policy-map 135 and class-map 104.

Again we can check the internal tables to identify those objects.

switch/Admin# show cfgmgr internal table policy-map | i 135
135             SLB                                 0      DATA_VALID,
switch/Admin# show cfgmgr internal table class-map | i 104
104             VIP-122-80                          0      DATA_VALID,

As you can see object 135 is policy-map “SLB” and object 104 is class-map “VIP-122-80”.  In my case, it does match the show service-policy command I initially typed.

If it does not match the rule you expected, it means ACE found a matching rule before the one you are testing.  Check your configuration and try to re-order the rules.

Don’t forget ACE looks into service policies sequentially and it stops when it finds a match.  So, it might not be the best match.  Try to order your rules accordingly.

If the rule is correct, but hit count is not incrementing, the traffic is probably dropped before it gets to the ACE device.  I will recommend reading the first post for troubleshooting connectivity issues.

Next time, I will focus on the other info contained in the show access-list trace command.

Gilles Dufour

Simon Chow
Cisco Employee

Great article, very often sniff trace is involve with troubleshooting load balancing issues, but the tips in this article certainly provide some quick checks before bringing in the sniff trace.

Kanwaljeet Singh
Cisco Employee

Hi Gilles,

Have i told you that you ROCK!!! If it didn't then i am telling you now:)

Great article.



Content for Community-Ad
This widget could not be displayed.