CSS 115001: maintenance best practice...

[gauravshar]

Hi there,

I need to know if there is any cisco guideline on scheduled-rebooting the CSS 115001 load balancer on regular basis.

In our environment, we have a CSS and after every 6-7 weeks, the applications behind it start giving problems. We used to reboot it every time to resolve this problem. Last time when this occurred, one of the servers behind the load balancer was found to be not responding to the users who were using the applications. The load balancer was showing the corresponding service UP but when we try to use the application using that server hostname/IP (not the load balancer IP), we were not able to reach it (rest of the servers were fine). The application team restarted/bounced the apache on that server and that resolved the problem at that time.

Although i have sufficient proof that the problem does not lie at Load balancer end, still I need to be sure why were able to resolve such problem by rebooting the load balancer previously. Does Cisco provide any guidelines on rebooting the CSS every after certain time to maintain the healthy loadbalanced connections.

Thanks and regards,

Gaurav

[Christopher Miles]

Hey Gaurav,

There is really no guideline for rebooting the CSS.. We've seen boxes stable and running for years without a problem, normally these types of problems are to do with the type of applications being load balanced and making sure configurations with timers and stickiness are set to accommodate the applications requirements.

My suggestion would be to do some troubleshooting when you see this issue next by way of sniffer traces both back and front of the css, this is the best way to understand what the application is doing and to make sure the CSS is configured correctly. I would also be looking to make sure connections have not been cleaned up and the counters are incrementing etc..

regards,

Chris

[gauravshar]

Hi Chris,

Thanks for the reply...

I am concerned about the total number of connection showing way beyond the allowed connections (65000+). I n the below output, they are showing in BOLD. The web console shows more than 100% of connections been utilised. This is what has been botherning the business/managers. Is it really something to get tense about .

Name: XYZ     Index: 2

Type: Local            State: Alive

Rule ( 1.2.7.26  TCP  8009 )

Session Redundancy: Disabled

Redirect Domain:

Redirect String:

Keepalive: (TCP-8009   5   3   5 )

Keepalive Encryption:      Disabled

Last Clearing of Stats Counters: 03/13/2011 10:50:35

Mtu:                       1500        State Transitions:            26

Total Local Connections:   827336      Total Backup Connections:     0

Current Local Connections: 10          Current Backup Connections:   0

Total Connections:         827336      Max Connections:              65534

Total Reused Conns:        0

Weight:                    1           Load:                         94

Weight Reporting:          None

I found that these Total Number of connections are the accumulated connection since the CSS' last reboot and with that definition it should not affect the existing load-balanced applications.. Please suggest.

Thanks,

Gaurav

[Christopher Miles]

Gaurav,

The Total Connections is the amount of connections this rule has received since reboot or zero-ing of the counters.. So in your case XYZ has received a total of 827336 connections since 03/13/2011 10:50:35, which I don't think is any cause for concern.

At the current point in time when you ran this command there was only 10 active connections. As connections like http are typically short lived the active connections could be 0 one second and 100 in the next.. So what you want to see is total connections continually incrementing as a sign XYZ is in fact receiving traffic. For long lived connections like FTP etc you would expect the connections to hang around for a much longer time.

The Max connections you highlighted is just advising you that service XYZ can accept a maximum of 65534 TCP connections, it is not saying there is currently 65534 connections.

What I do notice though is in the last 10 days this particular service has had 26 state transitions so at slightly more than twice a day the state of this service has bounced from Alive to Down .. Sometimes you might want to check check the the keepalive between the service and the server is at an interval that gives a solid response.

cheers,

Chris