CSS does not seem to be load balancing

[ahyaman718]

I am a CSS newbie, these switches set up before me starting here and no one knows how they work or are configured. Here is a sanitized configuration. The traffic is only going through "interim5". ANY help will be GREATLY appreciated as we are in a bind.

CSS11501# sh run
!Generated on 11/17/2010 08:42:37
!Active version: sg0810503

configure

!*************************** GLOBAL ***************************
  ip redundancy
  acl enable

  app
  app session zzz.zzz.1.2

  ftp-record TERMINAL xxx.xxx.11.171 USER des-password fffffffffffffffffffffff
  ftp-record MYTERMINAL xxx.xxx.11.177 CSSUSER des-password ffffffffffffffffffff

  ip route 0.0.0.0 0.0.0.0 xxx.xxx.11.170 1

!************************* INTERFACE *************************
interface e1
  bridge vlan 2

interface e4
  redundancy-phy

interface e8
  bridge vlan 10

!************************** CIRCUIT **************************
circuit VLAN1
  redundancy

  ip address xxx.xxx.11.173 255.255.255.248

circuit VLAN2

  ip address zzz.zzz.1.1 255.255.255.0
    redundancy-protocol

circuit VLAN10
  redundancy

  ip address xxx.xxx.11.179 255.255.255.248

!************************** SERVICE **************************
service interim4
  keepalive type tcp
  keepalive port 80
  ip address xxx.xxx.11.180
  active

service interim5
  keepalive type tcp
  keepalive port 80
  ip address xxx.xxx.11.181
  active

!*************************** OWNER ***************************
owner OFFICE

  content MIDDLE
    vip address xxx.xxx.11.169
    advanced-balance ssl
    add service interim4
    add service interim5
    active

!**************************** ACL ****************************
acl 1
  clause 1 permit tcp any destination xxx.xxx.11.169 eq 80
  clause 2 permit any xxx.xxx.11.169 destination any
  clause 3 permit any xxx.xxx.11.177 destination any
  clause 4 permit any xxx.xxx.11.178 destination any
  clause 5 permit any xxx.xxx.11.180 destination any
  clause 6 permit any xxx.xxx.11.181 destination any
  apply circuit-(VLAN1)

acl 2
  clause 1 permit any any destination any
  apply circuit-(VLAN2)

acl 10
  clause 1 permit any any destination any
  apply circuit-(VLAN10)

CSS11501#

[Marvin Rhoads]

The commands "show service summary" and "show rule OFFICE MIDDLE services" should give us some insight as to the state of the services and the rule to which they are assigned.

Please post the output of those commands.

BTW it appears you have a high availability pair. "show redundancy" should show you the status of that setup. Be advised that any configuration changes must be manually synchronized: "script play commit_redundancy "standy_unit_redundancy_interface_ip"" is the command you need to use (substituing the actual partner's IP but keeping it in quotes).

Also, if you're not running the most current system software, loading it is a good idea (assuming you have support). 8.20.4.02 is the latest release and it incorporates a number of bug fixes.

[ahyaman718]

Thanks for taking a look, here is the output of those commands

CSS11501# sh service summary

Service Name                     State     Conn  Weight  Avg   State
                                                         Load  Transitions

interim4                           Alive        10      1     2            0
interim5                           Alive        37      1     2            0

CSS11501# show rule OFFICE MIDDLE

Name:           MIDDLE   Owner:                    OFFICE
State:                 Active   Type:                     HTTP
Balance:          Round Robin   Failover:                  N/A
Persistence:          Enabled   Param-Bypass:         Disabled
Session Redundancy:  Disabled
IP Redundancy:    Not Redundant
L3:         xxx.xxx.11.169
L4:         Any/Any
Url:
Redirect: ""
TCP RST client if service unreachable: Disabled
Rule Services & Weights:
1: interim4-Alive, S-1
2: interim5-Alive, S-1
CSS11501#

[dpastor]

Your services are showing as active but the application running on them may not be working correctly, which may be giving you the indication of the CSS not load balancing.  The server can respond positively to TCP port 80 even if the application is down.

Dave

[ahyaman718]

A little more back story, one of the load balanced systems went down for a while, not sure if there is something thats needs to be done to reintroduce it back into the load balancing, but I suppose that would defeat the purpose of load balancing fault tolerance.

[Marvin Rhoads]

From the command output, both your services appear to be active and the rule is seeing them as such.

Can you share what indicator is telling you that you are only getting traffic on one of the real servers? The CSS appears to think that traffic is hitting both services. You can confirm this by successively entering the "sh rule OFFICE MIDDLE services" command and watching the hits increment. (Note the "services" parameter at the end of that command.)

Looking at your configuration, I am not sure why they put "advanced-balance ssl" in the rule. As a port 80 VIP I would assume you are running http, not https. According to the command reference, the usage of that command is for:

Enables the content rule to stick the client to the server
based on the Secure Sockets Layer (SSL) version 3
session ID assigned by the server. The application type
must be SSL for the content rule. You must specify a
port in the content rule to use this keyword. The CSS
will then spoof the connection.

[ahyaman718]

CSS11501#
CSS11501# show rule OFFICE MIDDLE services

Rule Services:
Local Load Threshold: 254
PrimarySorryServer: None
SecondSorryServer: None
VIP Ping Response Decision: Local Services Only

Name:            Hits:         Wgt:  State:   Ld: KAlive:   Conn: DNS:
-----            -----         ----- ------   --- -------   ----- ----
interim4           125,550,747   S-1   Alive    2   TCP-80    7     0
interim5           298,278,777   S-1   Alive    2   TCP-80    32    0

Location Cookie Services:
Name:            Hits:         State:   Ld: KAlive:   Conn:
-----            -----         ------   --- -------   -----

Last Clearing of Stats Counters:  11/16/2009 16:12:07

CSS11501#
CSS11501#

[dpastor]

drocks,

If you are concerned that the hit counts to the servers are not even, you could use the "balance leastconn" command on the content rule to help even things out.

interim4           125,550,747   S-1   Alive    2   TCP-80    7     0
interim5           298,278,777   S-1   Alive    2   TCP-80    32    0

Dave

[Marvin Rhoads]

Actually since those hit counts are cumulative (from the last CSS reboot, last zero or rule creation, as applicable) and the original poster mentioned the one server had been down for a while, that would be normal. That's why I suggested doing a reset of the counters and watching from that point forward. (Alternatively one could run the command repeatedly and expect to see increments in roughly equal amounts.)

[dpastor]

I agree.

Dave